Más contenido relacionado

Similar a IAU workshop 2018 day one(20)




IAU workshop 2018 day one

  1. Containers: Portable, repeatable user-oriented application delivery II HPC Saudi 2018 #dockerbday Christian Kniep @CQnib Walid Shaari @walidshaari
  2. AGENDA : Good Morning Containers
  3. $id Christian Over ten-year journey rooted in the industrial, automotive HPC in Germany, Christian started his career in Bull R&D supporting CAE applications and VR installations, then later Dyna. Co-founded the container and cloud workshop in ISC HPC conference when told at a meeting that HPC can not learn anything from the emerging Cloud and BigData companies. Since then, he is curious and leading DevOps and containerization effort wherever he goes. Just before Docker, he worked on the cloud-stack team at Sony PlayStation. Christian joined Docker Inc in 2017 to help push the adoption forward and be part of the innovation instead of an external bystander. During the day he helps Docker customers in the EMEA region to fully utilise the power of containers; at night he likes to explore new emerging trends by containerising them first and seek application in the nebulous world of DevOps. @kniepbert
  4. $id walid Passionate about Openness, Open Source, DevOps, Infosec Team member of the Expec Computer Center systems division Red Hat Certified Architect RHCA V Certified Kubernetes Administrator CKA SANS GIAC Incident handler, Forensics and Web security certified. Dhahran Docker & Ansible meetup organizer “Community Leader” @walidshaari
  5. Join the Docker Student Community! Sign up here: (with your school email) for access to our free Docker Student Developer Kit and more! Become a Docker Campus Ambassador! For leaders on campus who want to help their peers learn Docker! Learn more and apply here: Are you a student?
  6. Docker Community Leader! Docker Captain Docker Mentor Docker Active User What, If you are not?
  7. Let's get to know each other ▪ Assuming everyone knows a bit of ▪ Linux ▪ Unix ▪ Mac OSX CLI ? ▪ Development, Operations, Security, Research, Business, Others?  ▪ DevOps ▪ Containers ▪ Schedulers ▪ Containers ecosystem ▪ Clusters, Load balancers, Orchestration
  8. Goal Up and running with containers ecosystem informal interactive workshop format
  9. Happy 5th Birthday Docker! #dockerbday March 19-25, 2018
  10. Docker Bday #5 Celebrations Worldwide! 100+ events worldwide!
  11. Docker Momentum Thank You for 5 Amazing Years! Docker EE commercial customers 450+ Job listings on LinkedIn 15K Container downloads 37B 3.5M 200+ Active Docker user groups Dockerized apps
  12. Containers are the “Fastest Growing Cloud Enabling Technology” By 2020, more than 50% of global organizations will be running containers in production. -Gartner Title source: 451 Research 2017 24B PULLS
  13. Lab Instructions STEP 1: Visit Or Create Docker hub/store account: Join the Docker Community - Join the slack channel: #5th-bday #dockerbday
  14. STEP 2: Take a #dockerselfie #dockerbday
  15. © 2013-2016 Docker, Inc. All rights reserved HPC
  16. HPC or Scientific Computing? ▪HPC workloads mostly ▪ Runs on Linux ▪ Preferably on bare-metal for maximum performance, lower overhead ▪HPC Application ▪ Broken into smaller parallel distributed problems across a cluster of nodes. ▪ Utilizes interprocess communications heavily via shared memory, or across the network.
  17. HPC Status Quo ▪ HPC dominated by Academics research and discovery  ▪ Business HPC by the industry in the last 5-10 years seen an increase in HPC interest (Automotive, Finance, O&E) ▪ Possible constraints: ▪ Snowflake deployments, each HPC cluster/supercomputer is build in mind with specific use cases ▪ Long lived nodes. ▪ Bloated/drift/unclean maybe diskless reboots ▪ Reboot time, or launching app could be long due to system/memory checks, bootstrapping  ▪ Old Linux distribution ▪ Fixed installation based on single enterprise distro  (Scientific, RHEL)  ▪ Old kernel features
  18. HPC workload runs on the cloud 25%
  19. Which workloads and frameworks are running on OpenStack? Source: > 38% scientific/technical computing already happening on Openstack
  23. Container Technology 101 Traditional vs. Container Virtualization
  24. Stacked View Hardware Host Kernel Userland Services Hypervisor (type-2) Kernel Userland Services1 Services2 Userland Kernel Hardware Host Kernel Userland Services Userland appB appC Userland Cnt1 Cnt2 VM1 VM2 Traditional Virtualization os-virtualization VM-shortcuts (PVM,pci-passthrough)
  25. hardware hypervisor (type-1) container Traditional Virtualization kernel Interface View libs From Application to Kernel application libs application lib-calls 102 syscalls 101 hypercalls hardware kernel hw-calls os-virtualization
  26. Container Technology 101 Namespaces
  27. Namespaces Processes Isolation ● host sees all processes with real PID from the Kernels perspective ● first process within PID namespace gets PID=1 Host cnt0 ps -ef cnt1 java -jar .. cnt2 java -jar ..
  28. Resource Isolation of Process Groups 7 as of Kernel 4.10 1. MNT: Controls mount points 2. PID: Individual process table 3. NET: Network resources (IPs, routing,...) 4. IPC: Prevents the use of shared memory between processes 5. UTS: Individual host- and domain name 6. USR: Maps container UID to a different UID of the host 7. CGRP: Hides system cgroup hierarchy from container Other (incomplete list): ● RDMA ● Syslog ● Time
  29. Container Namespaces A starting container gets his own namespaces. PIDMNT IPCNET USR Host UTS CGRP cnt0 cnt1 cnt2 But can share namespaces with other containers or even the host
  30. Host All In When using all host namespaces - we are on the host (almost like ssh). PIDMNT IPCNET USRUTS CGRP cnt0 $ docker run -ti --rm --privileged --security-opt=seccomp=unconfined --pid=host --uts=host --ipc=host --net=host -v /:/host ubuntu bash root@linuxkit-025000000001:/# chroot /host / # ash / #
  31. Container Technology 101 cgroups / Layering Capabilities
  32. CGroups While namespaces isolate, Control Groups constraint resources.
  33. Overlay Filesystem Compose a FS from multiple pieces ubuntu:16.04 openjre:9-b114 appA.jar:1.1 appB.jar ARG FROM openjre:9-b114 COPY appB.jar /usr/local/bin/ CMD [“java”, “-jar”, “/usr/local/bin/appB.jar”] ARG FROM openjre:9-b114 COPY appA.jar /usr/local/bin/ CMD [“java”, “-jar”, “/usr/local/bin/appA.jar”] FROM ubuntu:16.04 ARG JRE_VER=9~b114-0ubuntu1 RUN apt-get update && apt-get install -y openjdk-9-jre-headless=${JRE_VER} && java -version openjre:9-b117
  34. First Step, toward a container definition? • What matters most? The application or data • The application can be a process or a set of processes • The use case might be not a running app • Set of tools  to develop an app • Set of scripts "apps" that are part of a pipeline • complete appliance • Isolated contained environment "Encapsulation" • Technical synonyms  • chroot • jail • partition • namespace • zone
  35. chroot/jail A chroot on Unix operating systems is an operation that changes the apparent root directory for the current running process and its children. A program that is run in such a modified environment cannot name (and therefore normally cannot access) files outside the designated directory tree. The term "chroot" may refer to the chroot(2) system call or the chroot(8) wrapper program. The modified environment is called a chroot jail.
  37. Thank the giants
  38. CONTAINERS? WHAT ARE THEY REALLY? Linux features? Namespace cgroupsLXC Union file systems Configuration management? Virtualization technology? npm jar Packaging ? rpm deb tar.gz Virtual/environment management ? Sandboxing? chroot BSD jail Solaris zones IBM VM/370 (1972) seccomp
  39. IT DEPENDS Manual Configuration Traditional VMs Less Portable Minimal overhead Most Portable Lots of overhead Configuration Management tools Containers Docker Intel Clear ContainersSingularity LXC/LXD Non-Repeatable Repeatable rkt
  40. Container Containment, isolation, or encapsulation of an environment. Machine container: Encapsulates a complete system image. e.g. Ubuntu, RHEL, Scientific Linux. Application container: Encapsulates a service/software. e.g. Django, ROR, Gitlab, redis, Openfoam, kafka, spark. what is the smallest application container?
  41. Possible HPC Caveats/Constraints 1. Memory/storage deduplication 2. Code Optimization for specific architecture 3. Limited take on HPC specific orchestration and scheduling 4. Hardware topology assumptions (e.g. GPU brand, interconnect) 5. Chroot based containers have none/limited tooling (e.g. introspection ) 6. Chroot based containers might be hard to scan for security vulnerabilities, hardening, and composition.
  42. DEVELOPERS LOVE DOCKER 42 451 Research
  44. Container Runtime docker < 1.11.0 └── systemd └── docker run OpenFoam └── Docker Engine └── OpenFoam docker > 1.11.0 └── systemd └── docker run OpenFoam └── Docker Engine └── containerd └── runc └── OpenFoam rkt > 1.0 └── systemd └── rkt run OpenFoam └── OpenFoam singularity (2.2.x) └── systemd/(init) └── bash └── OpenFoam
  45. Other runtime
  46. By the way:
  47. Image formats Layered Overlay filesystems/Graph drivers chrootDirectory Archive       #OCI #ACI
  48. Use Cases: Packaging Agnostic packaging Captures ○ Dependencies ○ Environment ○ Configurations ○ Executables ○ How about data? ○ What Else? ■ hint: m* Pack once, Run everywhere #EasyBuild #lmod #GUIX #NYU-Environment
  49. Use Case: Portability Portable/Scalable across ● platforms ● Distributions ● Environments Separation of concerns, e.g. development pack and ship, operations scale and deploy. development ensures app is resilient, operations enure infra is HA resilient and scalable
  50. Use Case: Portability Portable/Scalable across ● systems ● subsystems ● Anywhere
  51. Use Case: Reproducible Paolo Di Tommaso from the Center for Genomic Regulation presented : Manage Reproducibility of Computational Workflows with Docker Containers and Nextflow.
  52. Cloud use Case - Transport - Security CIA - at rest encrypted signed image - at runtime: - platform specific - scalability issues - PMIx to the rescue?!
  53. Data Center current state SchedulerScheduler Jobs Jobs Jobs Jobs Jobs Jobs Scheduler Jobs Jobs Jobs Cluster Management A Cluster Management B Cluster Management C
  54. Data Center Secure Allocation of Resources VC3 BigData VC1 Infra VC2 HPC SchedulerSchedulerScheduler DataCenter Scheduler jobs Jobs Jobs Jobs Jobs Jobs Jobs Jobs 2nd Generation Cluster Management
  55. Mesos ▪ Mature, Open Source Apache Project ▪ Cluster Resource Manager ▪ Scalable to over 10,000s of nodes ▪ Fault tolerant, no single point of failure ▪ Multi-tenancy with strong resource isolation ▪ Improved resource utilization
  56. Docker Performance
  57. NVIDIA Example use case
  58. MPI batch jobs ● use ssh inside container ● dssh ● Capitalize on openmpi ○ Openmpi/pbs/TORQUE ○ Process Management Interfaces PMIx ● Singularity examples uses Openmpi/Slurm ● mesos ● Commercial Univa support ● Research, and contribute ideas, pull requests to swarm, kubernetes, slurm, pbs pro ● Joing the HPC-SIG
  59. DISCLAIMER @kelseyhightower  : The problem with most blog posts attempting to compare two different systems is the author not having the sufficient experience to do so.
  60. © 2013-2016 Docker, Inc. All rights reserved 1. Introduction to Docker #dockerbday
  61. What is Docker? The leading open source platform to pack, ship and run apps as lightweight containers. Developers: use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators: use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises: use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux and Windows Server apps. #dockerbday
  62. • Standardized packaging for software and dependencies • Isolate apps from each other • Share the same OS kernel • Works for all major Linux distributions • Containers native to Windows Server 2016 What are Docker containers?
  63. Containers and VMs together Containers and VMs together provide a tremendous amount of flexibility for IT to optimally deploy and manage apps.
  64. Standards
  65. Docker Architecture Linux Container Implementation & Ecosystem Christian Kniep, v2018-01-18 Technical Account Manager
  66. Architecture on Linux Operating System Control Groups (cgroups) Namespaces (mnt,pid,ipc,...) Layer Capabilities AUFS,overlay,... Other OS Functionality Docker Engine REST interface libcontainerd libnetwork storage plugins containerd + runc Docker Client Docker Compose Docker Registry Docker Swarm/K8s
  67. Runtime runc + containerd ● ● containerd An industry-standard container runtime with an emphasis on simplicity, robustness and portability. ● runc CLI tool for spawning and running containers according to the OCI specification rootfs config.json runc executed container
  68. libnetwork Provide IP connectivity The goal of libnetwork is to deliver a robust Container Network Model that provides a consistent programming interface and the required network abstractions for applications.
  69. storage driver Handling OverlayFS The storage driver controls how images and containers are stored and managed on your Docker host.
  70. Plugins Extend Functionality of the Engine Framework to ‘intercept’ certain API calls and act on them. Current supported drivers: - VolumeDriver - NetworkDriver - IPAMDriver - LogDriver - MetricsCollector - Authentication (authz) // VolumeDriver type Driver interface { Create(Request) Response List(Request) Response Get(Request) Response Path(Request) Response Mount(Request) Response Unmount(Request) Response Capabilities(Request) Response }
  71. Architecture on Windows Operating System Other OS Functionality Docker Engine REST interface libcontainer libnetwork storage plugins Docker Client Docker Compose Docker Registry Docker Swarm/K8s Host Compute Service Control Groups Namespaces Layer Capabilities Object Namespace, Process Table, Networking Job Objects Registry, Union like filesystem extension
  72. Docker CE/EE
  73. Docker is the only Containers-as-a-Service platform for IT that manages and secures diverse applications across disparate infrastructure, both on-premises and in the cloud Multi-Architecture Operations Infrastructure Independence Secure Software Supply Chain COST SAVINGS Linux Mainframe AWS Azure Other Public Clouds Windows ENGINE FOR INNOVATION DOCKER ENTERPRISE EDITION
  74. Docker Enterprise Edition Capabilities Enterprise Edition Optimized Container Engine Integrated App and Cluster Management Certification and Support Policy Management Image Scanning and Monitoring Secure Access and User Management Content Trust and Verification Application and Cluster Management Image Management Security Distributed State Network Container Runtime Volumes Orchestration Application Composition, Deployment and Reliability Certified Containers Certified Plugins Certified Infrastructure
  75. © 2013-2016 Docker, Inc. All rights reserved Singularity From “Michael Bauer” Gent talks Fosdem/UoG EASYBuild
  76. Scientific computing container
  77. Singularity Container Selection Criteria
  78. Docker vs Singularity vs Shifter in an HPC environment
  79. © 2013-2016 Docker, Inc. All rights reserved rkt
  80. What is rkt? From the rkt GitHub page, "rkt (pronounced "rock-it") is a CLI for running app containers on Linux. rkt is designed to be secure, composable, and standards-based. #ACI
  81. Why rkt? ● Don’t want to run dockerd daemon. ● Don’t require the Docker’s rich feature set/ecosystem. ● Can’t trust Docker security yet.
  82. rkt # rkt run --interactive docker://ubuntu --insecure-options=image
  83. Thank you
  84. DOCKER HISTORY ▪ Started as internal project @ dotcloud ▪ Open Sourced in 2013 ▪ Developed in the open
  85. Forces and Motivations behind containers 90 Loosely Coupled Services Many Small Servers ~2000 Today Monolithic Big Servers Slow changing Rapidly updated