Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose

9.180 Aufrufe

Veröffentlicht am

Flink Forward 2015

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose

  1. 1. IBM Confidential 1 Apache Flink Cluster Deployment On Docker and Docker-Compose Simon Laws simon_laws@uk.ibm.com
  2. 2. © 2014 IBM Corporation What We’re Going To Build 2 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  3. 3. © 2014 IBM Corporation Docker 3 docker.com
  4. 4. © 2014 IBM Corporation Docker 4 docker.com/ whatisdocker
  5. 5. © 2014 IBM Corporation IBM Bluemix Sign up for a free account and experiment 5 bluemix.net
  6. 6. © 2014 IBM Corporation IBM Bluemix Signup Sign up for a free account and experiment Various options for academics, startups etc. Prices here for when you go over free allocation https://console.eu- gb.bluemix.net/pricing/ 6 https://console.ng.bluemix.net/registration/
  7. 7. © 2014 IBM Corporation Apache Flink and Docker 7 github.com/apache/ flink/tree/master/ flink-contrib/ docker-flink
  8. 8. © 2014 IBM Corporation Install Ubuntu Using VirtualBox 8 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  9. 9. © 2014 IBM Corporation Setting Up The Host Operating System  If you’re on Linux you’re probably good to go  If you’re on Windows or OSX you need to run Linux on a VM  As an alternative you could do this on a VM in the new VM service in Bluemix. It’s in Beta at the moment.  Get VirtualBox  Create a Linux VM (I used Ubuntu)  Install Docker  Download Flink Source  Try a Flink example 9
  10. 10. © 2014 IBM Corporation Get VirtualBox 10 virtualbox.org
  11. 11. © 2014 IBM Corporation Get Linux Distro (I went for Ubuntu 14.04) 11 releases.ubuntu.com/14.04
  12. 12. © 2014 IBM Corporation Create VM from ISO in VirtualBox (and set port forwarding) 12 End result is that I can now SSH into the instance of Ubuntu running inside VitualBox. Everything for this point on happens inside this Ubuntu VM To start with make sure that Ubuntu is up to date > sudo su - > apt-get update sudo su - apt-get updatesudo su - apt-get update
  13. 13. © 2014 IBM Corporation Install Docker and Flink 13 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  14. 14. © 2014 IBM Corporation Install Docker and Docker-Compose 14 github.com/apache/ flink/tree/master/ flink-contrib/ docker-flink
  15. 15. © 2014 IBM Corporation Download Flink Source 15 git clone https://github.com/apache/flink.git cd flink/flink-contrib/docker-flink The base Docker image source Builds the “base” and “flink” images The flink Docker image source Docker-Compose configuration
  16. 16. © 2014 IBM Corporation Docker Images and Containers 16 Source Docker Image Docker Container docker build docker run What Images Do I have? docker images What Containers Do I have running? Push elsewhere (Docker Hub) To share with others ssh or docker exec docker psDockerfile
  17. 17. © 2014 IBM Corporation Dockerfiles – The Recipe for a Docker Image 17
  18. 18. © 2014 IBM Corporation Build Docker Images 18 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  19. 19. © 2014 IBM Corporation Building The Images From the Docker Files 19 ./build.sh If you look in the script it use the “docker build” command to build a “base” image and a “flink” image This failed for me on the first attempt with the base build reporting chpasswd: (user root) pam_chauthtok() failed, error: System error chpasswd: (line 1, user root) password not changed It turned out I had an old version of Docker so I updated to latest and it worked
  20. 20. © 2014 IBM Corporation The Built Images 20
  21. 21. © 2014 IBM Corporation Run Docker Containers 21 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  22. 22. © 2014 IBM Corporation Running Linked Images Manually (To See What’s Going On) 22 docker run -d -p 48080:8080 -p 48081:8081 --expose 6123 --name jobmanager flink /usr/local/flink/bin/config-flink.sh jobmanager docker run -d --expose 6121 --expose 6122 --link jobmanager --name taskmanager1 flink The job manager Add a task manager This failed at first with a strange “Exec format exception”. I made a local fix by removing the comment from the top of the flink/config-flink.sh file leaving #!/bin/bash As the first line
  23. 23. © 2014 IBM Corporation Flink up and Running 23
  24. 24. © 2014 IBM Corporation Running a Test 24 I had previously uploaded flink-java-examples-0.9.1-WordCount.jar from my local Windows install of Flink
  25. 25. © 2014 IBM Corporation Accessing Containers with Docker Exec 25
  26. 26. © 2014 IBM Corporation Making Things Simpler With Docker-Compose 26 docker-compose up –d docker-compose scale taskmanager=3 docker-compose kill
  27. 27. © 2014 IBM Corporation Connecting To Bluemix 27 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  28. 28. © 2014 IBM Corporation Running the Images On Bluemix 28 Log in to Bluemix: 1.cf login Run the IBM Containers cf CLI plug-in. 2.cf ic login Create a Dockerfile in the root directory of your app source files. Learn more about Dockerfiles. Build an image from your Dockerfile. The command returns an image ID. 3. docker build -t image_name Tag the image with your private namespace in the IBM Containers registry. 4. docker tag image_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag The image name is optional. If it is not specified, the image is tagged with latest. Push this image to the IBM Containers registry: 5. docker push registry.stage1.ng.bluemix.net/sluk/image_name:image_tag You can create a container from this image in the Bluemix Catalog, or with the following command: 6. cf ic run --name container_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag
  29. 29. © 2014 IBM Corporation Install The Cloud Foundary Client Software 29 In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud Foundary client tool wget --output-document=cfcli.deb https://cli.run.pivotal.io/stable?release=debian64&version=6.12.4&source=githu b-rel dpkg -i cfcli.deb In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud Foundary client cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64
  30. 30. © 2014 IBM Corporation Log Into Bluemix – Web User Interface 30
  31. 31. © 2014 IBM Corporation Log Into Bluemix – Command Line Using CF Tool 31
  32. 32. © 2014 IBM Corporation Log Into Bluemix Containers – Command Line Using CF Tool 32
  33. 33. © 2014 IBM Corporation Push Image Up To Bluemix 33 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  34. 34. © 2014 IBM Corporation Tag and Push Up Our Images 34 docker tag flink registry.ng.bluemix.net/sluk/flink docker push registry.ng.bluemix.net/sluk/flink
  35. 35. © 2014 IBM Corporation List Images Now Up On Bluemix 35 cf ic images
  36. 36. © 2014 IBM Corporation Run Containers On Bluemix 36 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
  37. 37. © 2014 IBM Corporation Creating Containers in Bluemix via User Interface 37
  38. 38. © 2014 IBM Corporation Creating Containers in Bluemix via CF Command Line 38 Option 1) Leverage this Cloud Foundry IBM Containers plugin without affecting the local docker environment: Example Usage: cf ic ps cf ic images Option 2) Leverage the docker CLI directly. In this shell, override local docker environment to connect to IBM Containers by setting these variables, export DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443 export DOCKER_CERT_PATH=/root/.ice/certs export DOCKER_TLS_VERIFY=1 Example Usage: docker ps docker images
  39. 39. © 2014 IBM Corporation39 Create jobmanager Container in Bluemix via CF Command Line cf ic run -p 22:22 -p 8080:8080 -p 8081:8081 -p 6123:6123 --expose 6123 --name jobmanager registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config-flink.sh jobmanager Assign a public IP so you can get to the console
  40. 40. © 2014 IBM Corporation40 Create taskmanager Container in Bluemix via CF Command Line cf ic run -p 22:22 -p 6121:6121 -p 6122:6122 --expose 6121 -- expose 6122 --name taskmanager1 --link jobmanager:jobmanager registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config- flink.sh taskmanager
  41. 41. © 2014 IBM Corporation Running The Flink Cluster 41
  42. 42. © 2014 IBM Corporation QUESTIONS? 42

×