Lindsey Tulloch, Software Engineer Intern at Red Hat, presented 'The Salmon Algorithm Spawning with Kubernetes' at Eastern Canada's Kubernetes and Cloud Native Meetups in 2019. To see upcoming Kubernetes and Cloud Native meetups in Eastern Canada, please visit https://www.cloudops.com/workshop-calendar/#meetups

1. The Salmon Algorithm
Spawning with Kubernetes
Lindsey Tulloch

2. Lindsey Tulloch
Also likes: books, cats, coffee, bicycles, adventures
https://github.com/kubernetes-sigs/federation-v2
- Software Engineering Intern
@ Red Hat
- Multicluster May 2017 - Aug 2018
- Tekton Extension May 2019 - Thursday
- Kubernetes Release Team
- 1.12, 1.14, 1.15, 1.16 Release Notes
- Masters Student
@ University of Waterloo (as of next week!)

3. Spawning with Kubernetes
● Introduction and rationale
● Salmon
● Practical Considerations
● Containerization
● Compute Canada
● Implementation
● Experimental results
● Conclusion and Future
Work

4. Introduction & Rationale

5. Salmon

6. The Salmon Algorithm
● John Orth, 2012
● metaheuristic algorithm for combinatorial optimization problems inspired by:
○ Genetic Algorithms (population, generations, parent and child memories)
○ Ant Colony optimization (flow of water attracts salmon)
○ Salmon spawning behaviour*
Each salmon contains two lists:
- tabu list: vertices in the current path under construction
- memory list: copy of the parent's completed tabu list.
*The behaviour of the software salmon is idealized. We make no claim that real salmon behave in exactly this fashion.

7. The Salmon Algorithm
Previous work:
- Orth (2012): Error Correcting Codes, Code Clique Equivalence, TSP
- Orth, Houghten,Tulloch (2017) and Tulloch, Houghten (2017): Param-ILS to
tune Parameters, TSP, DNA Fragment Assembly
- Shows promising results for TSP
- might be useful on other combinatorial optimization problems.

8. DNA Fragment Assembly Problem to TSP
The layout phase of FAP can be translated into a TSP problem[19]
(Mallen-Fullerton,Fernandez-Anaya) with the cities representing fragments and
the distances representing overlap scores.
Two minor changes:
Negate Total: Minimization to Maximization
Remove distance to last city since FAP is not circular (insertion of dummy city is
one solution)

9. Why Kube?
(if it ain’t broke. . .)
- Increased use of containers (including
Singularity, Charliecloud, Docker).
- Research moving away from strict “job” style
workflows.
- Adoption of data-streaming and in-flight
processing.
- Greater use of interactive Science Gateways.
- Dependence on other more persistent
services--not aligned with HPC systems
- Increasing demand for reproducibility.
(if it ain’t broke. . .)
optimal, fix it!

10. 10
Advantages:
- Same API across ALL cloud resources with predictable results
- Promise of gitops: abstract away the running and processing of
research jobs
- Very popular, lots of talented engineers working on it, thriving
community
- Long-term outlook looks good
Does Kubernetes make sense as a research platform?

11. Practical Considerations

12. Compute Canada
Regional and Government Partners

14. Compute Canada
● Not-for-profit corporation
● Membership includes most of Canada’s
major research universities
● All Canadian faculty members have access to Compute Canada systems and can sponsor
others:
- students
- postdocs
- external collaborators
● No fee for Canadian university faculty
● Reduced fee for federal laboratories and
not-for-profit orgs

15. Compute Canada
● Compute and storage resources, data centres
● Team of ~200 experts in utilization of advanced
research computing
● 100s of research software packages
● Cloud compute and storage (openstack, owncloud)
● 5-10 Data Centres
● 300,000 cores
● 12 Pflops, 50+ PB

16. Compute Canada
Researchers drive innovation
● The CC user base is broadening,
bringing a broader set of needs.
● Tremendous interest in services
enabling Research Data Management
(RDM)

17. Containerization

18. Docker
This is Docker This is Docker on HPC

19. Alternatives

20. 20
Advantages:
- Same API across ALL cloud resources with predictable results
- GitOps
- Very popular,
Disadvantages:
- Complete overhaul of existing systems (Kubernetes and HPC
are not compatible at present)
- Huge learning curve - + yaml, docker, github, argo, kubeflow
- Desert of Expertise in academia
- Concern that corporate open source may not be as friendly for
academia as we’d like to think
Does Kubernetes make sense as a research platform?

21. Implementation

22. Salmon on Kubernetes
● Arbutus Cloud Project Access
○ Openstack
○ Maximum Resource Allocation
■ 5 Instances, 16 VCPUs, 36GB RAM, 5 Volumes, 70GB Volume Storage
■ 5 Floating IPs, 6 Security Groups
● Deploy Kubernetes with Kubespray, Terraform and Ansible
● Containerize the Salmon Algorithm
● Create Argo workflow
Salmon on Kubernetes

23. Salmon on KubernetesArbutus/Openstack

25. Salmon on KubernetesQuay Container Registry

26. Salmon on KubernetesContainer Logic

27. Salmon on KubernetesWorkflow Logic
YAML
YAML

28. Salmon runs with Argouns

29. Salmon resultsSalmon Results

30. Experimental Results

31. Ease of
Deployment
For a 12 month project. . .
August
- Started preliminary brainstorming ahead of time
September
- Official project start
October
- Meetings with Compute Canada staff interested
in Kubernetes
November
- Request RAS
- End of November--actual RAS allocation

32. Ease of
Deployment
For a 12 month project. . .
December
- Kubecon Seattle, meet Compute Canada staff
working on Kubernetes F2F
- Ryan Taylor (@ Compute Canada) and Bob Killen
(@ U of Michigan) offer support
January
- Figuring out how to get Kubernetes up and
running on Openstack
February
- Need more resources than originally requested
- Still working through oddities in the Kubernetes
deployment

33. Ease of
Deployment
For a 12 month project. . .
March
- Kubernetes deployed!!! 🙌🙌🙌🙌
April
- Containerizing the salmon algorithm for TSP
- Figuring out and deploying Argo workflows
May
- Argo workflows successful
- DNA Fragment Assembly containers
- Salmon container git repo with CI
June
- Cluster Flakes due to maintenance
- Paper writing

34. Ease of
Deployment
For a 12 month project. . .
July
- Paper writing and submissions
- Presentation
- Graduate. . .?
NO KUBERNETES
FOR 7/11 MONTHS

35. Conclusions & Future Work

36. Future of Kubernetes at CC
○ Learning curve is steep and time is precious (installing Kubernetes on bare
metal just to run your workflow is not worth it)
○ Lack of expertise with essential tools (yaml, docker, github)
○ Given an existing Kubernetes cluster, with a knowledgeable admin that can
assist you--Kubernetes offers a lot of benefits worth exploring
■ Completely automated, reproducible research workflows
■ Tools optimized for containerized, persistent services
■ Dependency hell eliminated
■ Roll back of Kubernetes versions can be fairly painless
To Kube or not to Kube?

37. Providers
● Offer Kubernetes for people to consume
● Get involved with the Kube community
● Learn as much as you can
● Provide outreach to researchers and
anyone that might need to be ramped up

38. Researchers
● Engage with research institutions
● Get involved with the Kube community
● Learn as much as you can
● Share your findings widely!

39. Future Research
- Performance Comparisons
- Git ops
- Syslabs.io’s “Slurm operator”
- Best practices for deploying
Kubernetes on prem

40. Thank You!

41. Useful Links
● CNCF Research User Group
● CNCF Academic Mailing List
● CNCF Academic Slack (#academia)
● Batch Jobs Channel (#kubernetes-batch-jobs)
● Kubernetes Big Data User Group
● Kubernetes Machine Learning Working Group

42. References
[1] G. M. Kurtzer, V. Sochat, and M. W. Bauer, “Singularity: Scientific containers for mobility of compute,” PLOS ONE, vol. 12, no. 5,
p. e0177459, May 2017. [Online]. Available: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459
[2] R. Priedhorsky and T. Randles, “Charliecloud: Unprivileged containers for user-defined software stacks in HPC,” p. 12.
[3] B. Burns, “The History of Kubernetes & the Community Behind It,” 2018. [Online]. Available:
https://kubernetes.io/blog/2018/07/20/
the-history-of-kubernetes--the-community-behind-it/
[4] B. Burns, B. Grant, D. Oppenheimer, E. Brewer, and J. Wilkes, “Borg, Omega, and Kubernetes,” p. 24.
[5] “CERN Case Study,” jun 2019. [Online]. Available: https://kubernetes.io/case-studies/cern/
[6] K. Sheets and S. Telfer, “Kubernetes, HPC and MPI,” Nov. 2018. [Online]. Available: https://www.stackhpc.com/k8s-mpi.html
[7] A. Ingersoll, “The full-time job of keeping up with Kubernetes,” https://gravitational.com/blog/kubernetes-release-cycle/. [Online].
Available: https://gravitational.com/blog/kubernetes-release-cycle/
[8] J. Orth, S. Houghten, and L. Tulloch, “Evaluation of the salmon algorithm,” in 2017 IEEE Conference on Computational
Intelligence
in Bioinformatics and Computational Biology (CIBCB), Aug. 2017, pp. 1–8.
[9] J. Orth, “The Salmon Algorithm-A New Population Based Search Metaheuristic,” Master’s Thesis, Brock University, 2012.
[Online].
Available: https://dr.library.brocku.ca/handle/10464/3929
[10] C. Pahl, A. Brogi, J. Soldani, and P. Jamshidi, “Cloud Container Technologies: a State-of-the-Art Review,” IEEE Transactions
on Cloud
Computing, vol. PP, pp. 1–1, May 2017.
[11] “Open Container Initiative, About,” https://www.opencontainers.org/faq, url = https://www.opencontainers.org/faq, language =
en-US, urldate = 2019-07-01, author = The Linux Foundation, journal = Open Containers Initiative, file = Snap-
shot:/home/banana/.zotero/zotero/7pdexnl0.default/zotero/storage/CJNUMT6R/faq.html:text/html.

43. References
[12] S. Telfer, “The State of HPC Containers,” https://www.stackhpc.com/the-state-of-hpc-containers.html. [Online]. Available:
https://www.stackhpc.com/the-state-of-hpc-containers.html
[13] C. Boettiger, “An introduction to Docker for reproducible research, with examples from the R environment,” ACM SIGOPS
Operating
Systems Review, vol. 49, no. 1, pp. 71–79, Jan. 2015, arXiv: 1410.0846. [Online]. Available: http://arxiv.org/abs/1410.0846
[14] “What is Kubernetes,” https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/. [Online]. Available:
https://kubernetes.io/docs/
concepts/overview/what-is-kubernetes/
[15] B. Gruening, O. Sallou, P. Moreno, F. da Veiga Leprevost, H. Ménager, D. Søndergaard, H. Röst, T. Sachsenberg, B. O’Connor,
F. Madeira, V. Dominguez Del Angel, M. R. Crusoe, S. Varma, D. Blankenberg, R. C. Jimenez, BioContainers Community, and
Y. Perez-Riverol, “Recommendations for the packaging and containerizing of bioinformatics software,” F1000Research, vol. 7, p.
742,
Jun. 2018. [Online]. Available: https://f1000research.com/articles/7-742/v1
[16] D. M. Jacobsen and R. S. Canon, “Contain This, Unleashing Docker for HPC,” p. 8, 2015.
[17] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at Google with
Borg,” in Proceedings of the Tenth European Conference on Computer Systems - EuroSys ’15. Bordeaux, France: ACM Press, 2015,
pp. 1–17. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2741948.2741964
[18] Hutter, F., Hoos, H.H., LeytonBrown K., Stuzle, T., “ParamILS: An Automatic Algorithm Configuration Framework,” Journal
of Artificial Intelligence Research, vol. 36, pp. 267–306, Oct. 2009. [Online]. Available: http://www.cs.ubc.ca/labs/beta/Projects/
ParamILS/papers/09-JAIR-ParamILS.pdf

44. References
[19] G. M. Mallén-Fullerton, J. A. Hughes, S. Houghten, and G. Fernández-Anaya, “Benchmark datasets for the dna fragment
assembly problem,” International Journal of Bio-Inspired Computation, vol. 5, no. 6, pp. 384–394, 2013. [Online]. Available:
http://www.inderscienceonline.com/doi/abs/10.1504/IJBIC.2013.058912
[20] R. M. Karp, “Reducibility among combinatorial problems,” in Complexity of Computer Computations: Proc. of a Symp. on the
Complexity of Computer Computations, ser. The IBM Research Symposia Series, R. E. Miller and J. W. Thatcher, Ed. New York,
NY: Plenum Press, 1972, pp. 85–103. [Online]. Available: http://cgi.di.uoa.gr/ ∼ sgk/teaching/grad/handouts/karp.pdf
[21] L. Tulloch and S. Houghten, “The Salmon Algorithm Return of the Parameters,” 2017.
[22] Gerhard Reinelt, “TSPLIB.” [Online]. Available: http://comopt.ifi.uni-heidelberg.de/software/TSPLIB95/
[23] J. A. Hughes, S. Houghten, and D. Ashlock, “Recentering and restarting a genetic algorithm using a generative representation
for an ordered gene problem1,” International Journal of Hybrid Intelligent Systems, vol. 11, no. 4, pp. 257–271, Sep. 2014. [Online].
Available: http://www.medra.org/servlet/aliasResolver?alias=iospress&doi=10.3233/HIS-140198[24] G. M. Mallén-Fullerton and G.
Fernandez-Anaya, “DNA fragment assembly using optimization,” in Evolutionary computation (CEC),
2013 IEEE congress on. IEEE, 2013, pp. 1570–1577. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/6557749/
[25] J. Hughes, “A Study of Ordered Gene Problems Featuring DNA Error Correction and DNA Fragment Assembly with a Variety
of Heuristics, Genetic Algorithm Variations, and Dynamic Representations,” Master’s Thesis, Brock University, 2014. [Online].
Available: http://www.dr.library.brocku.ca/handle/10464/5600
[26] R. J. Parsons, S. Forrest, and C. Burks, “Genetic algorithms, operators, and DNA fragment assembly,” Machine Learning, vol.
21, no. 1, pp. 11–33, 1995. [Online]. Available: http://www.springerlink.com/index/P892188783H43M04.pdf
[27] A. Hughes, James, S. Houghten, and D. Ashlock, “Recentering and restarting a genetic algorithm using a generative
representation for an ordered gene problem,” International Journal of Hybrid Intelligent Systems, vol. 11, pp. 257–271, 2014.
[28] L. Tulloch, “Deploying research workflows with Argo on Openstack Kubernetes clusters through compute canada:
onyiny-ang/4f90-kubernetes,” Jul. 2019. [Online]. Available: https://github.com/onyiny-ang/4F90-kubernetes

45. References
[29] ——, “Base code and container build repository for the Salmon Algorithm and basic DNA Fragment Assembly benchmark
problems: onyiny-ang/salmon,” Jul. 2019, original-date: 2019-05-10T05:53:00Z. [Online]. Available:
https://github.com/onyiny-ang/salmon
[30] S. Team, “Introducing HPC Affinities to the Enterprise: A New Open Source Project Integrates Singularity and Slurm via
Kubernetes,” May 2019. [Online]. Available: https://www.sylabs.io/2019/05/
introducing-hpc-affinities-to-the-enterprise-a-new-open-source-project-integrates-singularity-and-slurm-via-kubernetes/
[31] F. da Veiga Leprevost, B. A. Grüning, S. Alves Aflitos, H. L. Röst, J. Uszkoreit, H. Barsnes, M. Vaudel, P. Moreno, L. Gatto,
J. Weber, M. Bai, R. C. Jimenez, T. Sachsenberg, J. Pfeuffer, R. Vera Alvarez, J. Griss, A. I. Nesvizhskii, and Y. Perez-Riverol,
“BioContainers: an open-source and community-driven framework for software standardization,” Bioinformatics, vol. 33, no. 16, pp.
2580–2582, Aug. 2017. [Online]. Available: https://academic.oup.com/bioinformatics/article/33/16/2580/3096437
[32] J. A. Novella, P. Emami Khoonsari, S. Herman, D. Whitenack, M. Capuccini, J. Burman, K. Kultima, and O. Spjuth,
“Container-based bioinformatics with Pachyderm,” bioRxiv, Apr. 2018. [Online]. Available:
http://biorxiv.org/lookup/doi/10.1101/299032
[33] O. Spjuth, M. Capuccini, M. Carone, A. Larsson, W. Schaal, J. Novella, P. Di Tommaso, C. Notredame, P. Moreno, P. E.
Khoonsari, S. Herman, K. Kultima, and S. Lampa, “Approaches for containerized scientific workflows in cloud environments with
applications in life science,” August 2018. [Online]. Available: https://peerj.com/preprints/27141
[34] S. Turol, C. Gutierrez, and S. Matykevich, “A Multitude of Kubernetes Deployment Tools: Kubespray, kops, and kubeadm,” https:
//www.altoros.com/blog/a-multitude-of-kubernetes-deployment-tools-kubespray-kops-and-kubeadm/, Jun. 2018. [Online]. Available:
https://www.altoros.com/blog/a-multitude-of-kubernetes-deployment-tools-kubespray-kops-and-kubeadm/