OpenNebula out in the Open, Ander Astudillo, SURFsara

•

1 gefällt mir•455 views

Where OpenNebula is typically used by system administrators to manage data centers, SURFsara’s HPC Cloud exposes OpenNebula’s web interface crudely to end users. SURFsara’s end users are researchers. They all have diverse needs and most of them are no system experts. It turns out that OpenNebula’s interface is easy enough to use, enabling researchers to build complex compute infrastructures for their analysis workflows. This presentation discusses the features our researchers and we appreciate in OpenNebula, it pictures a selection of succesful projects and it also explains caveats we have come across while keeping the service in the air.

Technologie

— IaaS in the real world
OpenNebula out in the Open
Ander Astudillo <ander.astudillo@surfsara.nl>
Ede (The Netherlands), 19th September 2017
OpenNebula Tech Day

Ander Astudillo <ander.astudillo@surfsara.nl>
Science Park, Amsterdam
The SURF cooperative

What do we (SURFsara) want to offer?
3
Services for scientists
…scientists systems gurus
… complex users’ problems
• Data: big, dirty, non-structured…
• Computation: complex (e.g.: modeling, simulation)
• Libraries nightmare
• 3rd party, incompatibility, maintenance…
… flexibility
… trial and error
… test
… cooperate
… share
… show
… scratch … privacy

Agenda
1.- SURFsara & HPC Cloud
2.- The real world
Appendix.- SURFsara’s implementation

HPC Cloud IaaS ≡ Your craft
61.- SURFsara & HPC Cloud
Your application
may need more…
Scale up Scale outvs.

The prelude
71.- SURFsara & HPC Cloud
Claudia: Exploring the concept
• Couple of borrowed nodes
• 16 cores
• 1Gbps network
• Local storage
• Self-built UI
• vs. OpenNebula 1.x !

Calligo
81.- SURFsara & HPC Cloud
HPC
• Big nodes (19×)
• 32 cores, 256GB RAM
• Fast interconnect
• Low-latency 40 Gbps
• NFS storage • Web interface
• Groups
Cloud
• Multi-purpose versatility
• Shape elasticity
• Self-service on-demand
• Project-based
• Private network
Service
!
3.x

Rounding the system off
102.- The real world
• Documentation
• System load
• Support
• Accounting
Cloud light
• Smaller nodes
• CPU overcommit
• (Interactive) services
…scientists sysadmins
• Image upload
• Network filters
• DHCP
• Dynamic DNS
• Wizard
tweaks

Storage
112.- The real world
• Fast I/O
• More nodes
Local SSD
VirDir
• Per-project NFS share
• Image upload & download
• Backups
MySQL aaS
Archive
Data Ingest (DIS)

Incidents & eventualities
122.- The real world
Datacenter power outage
• Storage collapsed
• Helped discover power
redundancy was faulty
Enthusiastic users
• Component down
• Uncovered lenient quotas
‘s glory
Delete images
• Deleted users’ data
• Uncovered bug
VNC ports
• Ran out of ports
• Uncovered limitation
Too many tweaks
• Could not upgrade
• Uncovered our optimism

The present
132.- The real world
• Latest release of OpenNebula (v.5.2)
• Accounting
• Ceph storage; expansion
• Distributed object store and file system
• Cope with increasing load
• GPU processing
• Highly parallel structure
• Program specifically to use it
Oort: Consolidating
the concept

14
The present… eventualities
2.- The real world
SSD disks
• Consumer grade
• Helped sharpen our tenders
‘s upgrade glory
Inter-version UI changes
• Users’ VM’s logs
• Terminate vs. Delete
• ACL!
Resources bookkeeping
• Double VNC ports assignment
• Double GPU’s assignment
Datacenter move
• Remember cables!
• Cloud-init?

User applications
152.- The real world
• Flexible software mix
• Big VMs
• Elasticity
• Provide their own service to
their own users
• Software that requires licenses
• Set up, test and deploy workflows
• Deliver training; courses
• Intensive computing
•
…from diverse fields:
• Biology
• Genetics
• Informatics
• Chemistry
• Ecology
• Linguistics
• Robotics
• Business
• Social sciences
• Engineering
• Humanities
• Water management
• …
Users like& leverage…
‘s glory

SURFsara’s HPC Cloud
implementation
ppendix

Ander Astudillo
<ander.astudillo@surfsara.nl>
Credits
Images: Wikipedia, Science Park, RRZE icons,
NIST, nVidia, Ceph, Pixabay
Slides: SURFsara colleagues

Weitere ähnliche Inhalte

Was ist angesagt?

Open stack

Oded Rotter

Wido den Hollander - building highly available cloud with Ceph and CloudStack

ShapeBlue

Audience Level All levels Synopsis Peter has been involved in OpenStack community since its B-release, and he has been enabling and helping customers across various industries adopt OpenStack in strategic ways. In this session, you will learn from his experience what Red Hat’s perspective is on the current state of affairs in the OpenStack community and the path we see ahead that Red Hat is putting its efforts in. OpenStack is not a product that tries to solve any one business problem in particular, but a technology that aims to be usable for many – what are the required steps to make sure that your organisation is ready for the OpenStack-based cloudification and transformation. Speaker Bio: Peter Jung is a Senior Business Development Manager at Red Hat where he leads the practice in the areas of Cloud, SDN/NFV and IoT across Australia and New Zealand. He is passionate about open innovation and open source software development model as the foundation for next generation society and ICT systems. Prior to Red Hat, he had various roles at Cisco and Dell for 15 years. He holds a BSEE and an MBA. OpenStack Australia Day Melbourne 2017 https://events.aptira.com/openstack-australia-day-melbourne-2017/

OpenStack and Red Hat: How we learned to adapt with our customers in a maturi...

OpenStack

OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...

OpenNebula Project

7 distributed storage_open_stack

openstackindia

ARM server, The Cy7 Introduction by Aaron Joue, Ambedded Technology

Aaron Joue

OpenNebula Conf 2014 | ONE BIT to rule them all - Stefan Kooman

NETWAYS

Open Source vs. Open Standards by Sage Weil

Red_Hat_Storage

OTechs Cloud Computing Training Course

Osman Suliman

Software Defined Storage Appliance Power by ARM based Microserver

Aaron Joue

OpenStack Meetup Amsterdam 17-4-2014

zokahn

Business Intelligent

Tuan Luong

From Linux Foundation Collaboration Summit 2015, as presented by Larry Wikelius of Cavium Inc on 2015 02-18. ARM-powered servers for cloud computing and data center servers took a big step forward when Cavium recently announced availability of the industry’s first 48-core family of ARMv8 workload optimized processors. Delivering the highest core counts and most comprehensive set of I/O and accelerators in the market, ThunderX-based solutions are fully optimized for a targeted set of highly scalable workloads. In his talk, Larry Wikelius, who spearheads ecosystems and partner enablement for Cavium, will discuss growing momentum for ARMv8-based servers for hyperscale data center, cloud servers, big data and scale out computing as well as share new performance benchmark data. In addition, developments in the networking and carrier space are quickly mobilizing behind programs like The Open NFV Organization. The Linux Foundation, Xen Project, Open Compute Project and Linaro are also key.

LF Collab Summit 2015: ARM Servers for the Next Generation Date Center and Cl...

The Linux Foundation

Energy Saving ARM Server Cluster Born for Distributed Storage & Computing

Aaron Joue

Sas a peek_into_the_future_marty_czekalski

scsibeast

The platform currently runs several solutions for Earth Sciences Researchers: Developer Cloud Sandboxes for scalable scientific processors integration, Virtual Archives federating distributed data repositories, Data Challenges for Earth Observation contests, and Digital Marketplaces for reproducible scientific experiments. OpenNebula is also powering our Cloud development environment, that has been enhanced during the past year. Terradue’s Cloud development environment is both our laboratory to test the latest developments from OpenNebula, especially for the integration of our own OpenNebula extensions, and our Engineering team facility to provision servers supporting project-based software developments. OpenNebula provides the virtualization and management of hardware clusters, that we rent from commercial ‘bare-metal’ providers. We have recently further developed several specific drivers for Multi-Cloud bursting, in order to provision virtual machines over public commercial clouds. When their processor integration and validation phase concludes, our researcher users can seamlessly burst their applications at scale, leveraging OpenNebula drivers for on-demand processing tasks.

OpenNebula Conf 2014: Expanding OpenNebula´s support for Cloud Bursting - Emm...

NETWAYS

Alluxio Use Cases at Strata+Hadoop World Beijing 2016

Alluxio, Inc.

Leveraging Alluxio with Spark SQL to Speed Up Ad-hoc Analysis

Wenchun Xu

Mosix Cluster

Abhay Pai

C* Summit EU 2013: Ontorion: Scalable Information Management

DataStax Academy

Was ist angesagt? (20)

Open stack

Wido den Hollander - building highly available cloud with Ceph and CloudStack

OpenStack and Red Hat: How we learned to adapt with our customers in a maturi...

OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...

7 distributed storage_open_stack

ARM server, The Cy7 Introduction by Aaron Joue, Ambedded Technology

OpenNebula Conf 2014 | ONE BIT to rule them all - Stefan Kooman

Open Source vs. Open Standards by Sage Weil

OTechs Cloud Computing Training Course

Software Defined Storage Appliance Power by ARM based Microserver

OpenStack Meetup Amsterdam 17-4-2014

Business Intelligent

LF Collab Summit 2015: ARM Servers for the Next Generation Date Center and Cl...

Energy Saving ARM Server Cluster Born for Distributed Storage & Computing

Sas a peek_into_the_future_marty_czekalski

OpenNebula Conf 2014: Expanding OpenNebula´s support for Cloud Bursting - Emm...

Alluxio Use Cases at Strata+Hadoop World Beijing 2016

Leveraging Alluxio with Spark SQL to Speed Up Ad-hoc Analysis

Mosix Cluster

C* Summit EU 2013: Ontorion: Scalable Information Management

Ähnlich wie OpenNebula out in the Open, Ander Astudillo, SURFsara

SURFsara Cloud - OpenNebula Tech Day 2014.03.26

OpenNebula Project

CloudLab Overview

Ed Dodds

Cloud Computing Berkeley.pdf

AtaulAzizIkram

Cloud Computing from Academic Perspective

Dedy Hariyadi

20181219 ucc open stack 5 years v3

Tim Bell

20181219 ucc open stack 5 years v3

Tim Bell

e-Infrastructure available for research, using the right tool for the right job

David Wallom

Cloud computing course introduction

Haddy El-Haggan

Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...

Ceph Community

Infinitely Scalable Clusters - Grid Computing on Public Cloud - London

Hentsū

_Cloud_Computing_Overview.pdf

TyStrk

Week 1 Lecture_1-5 CC_watermark.pdf

John422973

Week 1 lecture material cc

Ankit Gupta

vssutcloud computing.pptx

MunmunSaha7

Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...

Larry Smarr

Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...

Larry Smarr

Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...

Larry Smarr

Hadoop As The Platform For The Smartgrid At TVA

Cloudera, Inc.

Big data nyu

Edward Capriolo

Cloud101-Introduction to cloud

Ranjan Ghosh

Ähnlich wie OpenNebula out in the Open, Ander Astudillo, SURFsara (20)

SURFsara Cloud - OpenNebula Tech Day 2014.03.26

CloudLab Overview

Cloud Computing Berkeley.pdf

Cloud Computing from Academic Perspective

20181219 ucc open stack 5 years v3

e-Infrastructure available for research, using the right tool for the right job

Cloud computing course introduction

Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...

Infinitely Scalable Clusters - Grid Computing on Public Cloud - London

_Cloud_Computing_Overview.pdf

Week 1 Lecture_1-5 CC_watermark.pdf

Week 1 lecture material cc

vssutcloud computing.pptx

Panel: Open Infrastructure for an Open Society: OSG, Commercial Clouds, and B...

Hadoop As The Platform For The Smartgrid At TVA

Big data nyu

Cloud101-Introduction to cloud

Mehr von OpenNebula Project

We've made our way into the world of open cloud — where each organization can find the right cloud for its unique needs. A single cloud management platform cannot be all things to all people. There will be a cloud space with several offerings focused on different environments and/or industries. The OpenNebula commitment to the open cloud is at the very base of its mission — to become the simplest cloud enabling platform — and its purpose — to bring simplicity to the private and hybrid enterprise cloud. OpenNebula exists to help companies build simple, cost-effective, reliable, open enterprise clouds on existing IT infrastructure. The OpenNebula Conference will be a great opportunity to communicate and share our vision and commitment, to look back at how the project has grown in the last 9 years, and to shed some insight into what to expect from the project in the near future.

OpenNebulaConf2019 - Welcome and Project Update - Ignacio M. Llorente, Rubén ...

OpenNebula Project

Computer networks are undergoing a phenomenal growth, driven by the rapidly increasing number of nodes constituting the networks. At the same time, the number of security threats on Internet and intranet networks is constantly increasing, and the testing and experimentation of cyber defense solutions require the availability of separate, test environments that best reflect the complexity of a real system. Such environments support the deployment and monitoring of complex mission-driven network scenarios, and cyber security training activities, thus enabling enterprises to study cyber defense strategies and allowing security researchers to evaluate their algorithms at scale. The main objective is delivering to researchers and practitioners an overview of the technological means and the practical steps to setup a private cloud platform based on OpenNebula for the creation and management of virtual environments that support cyber-security activities of training and testing, as well as an overview of its possible applications in the cyber security domain. In particular: 1. We describe our infrastructure based on OpenNebula 2. We overview our application, sitting on top of OpenNebula, as well as the technological tools involved in the management of its lifecycle (e.g., Ansible) . 3. We show how the platform can support various examples of security research activities  [References] Building an emulation environment for cyber security analyses of complex networked systems, Tanasache, Florin Dragos and Sorella, Mara and Bonomi, Silvia and Rapone, Raniero and Meacci, Davide, ICDCN '19, ACM, 2019

OpenNebulaConf2019 - Building Virtual Environments for Security Analyses of C...

OpenNebula Project

OpenNebulaConf2019 - CORD and Edge computing with OpenNebula - Alfonso Aureli...

OpenNebula Project

OpenNebulaConf2019 - 6 years (+) OpenNebula - Lessons learned - Sebastian Man...

OpenNebula Project

OpenNebula users have a range of storage options available to them, including proprietary appliances, proprietary software and Open Source software projects. This session will present a fully Open Source approach, that tightly integrates with Linux, and makes full use of the mature building blocks within the Linux kernel (LVM, Software RAID, DM-crypt, NVMe-oF Target, DRBD, etc...), and delivers one of the highest performance open source storage stacks currently available. The core goal is to expose the improved performance of NVMe storage devices to VMs and containers. The solution covers both local NVMe drives and NVMe-oF. For interacting with NVMe-oF targets it supports the Swordfish-API and LVM & Linux’s software NVMe-oF target. The solution contains a storage addon for OpenNebula.

OpenNebulaConf2019 - Performant and Resilient Storage the Open Source & Linux...

OpenNebula Project

OpenNebulaConf2019 - Image Backups in OpenNebula - Momčilo Medić - ITAF

OpenNebula Project

At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud. I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.

OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Ph...

OpenNebula Project

OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...

OpenNebula Project

Replacing vCloud with OpenNebula

OpenNebula Project

NTS: What We Do With OpenNebula - and Why We Do It

OpenNebula Project

OpenNebula from the Perspective of an ISP

OpenNebula Project

NTS CAPTAIN / OpenNebula at Julius Blum GmbH

OpenNebula Project

Performant and Resilient Storage: The Open Source & Linux Way

OpenNebula Project

NetApp Hybrid Cloud with OpenNebula

OpenNebula Project

NSX with OpenNebula - upcoming 5.10

OpenNebula Project

Security for Private Cloud Environments

OpenNebula Project

CheckPoint R80.30 Installation on OpenNebula

OpenNebula Project

DE-CIX: CloudConnectivity

OpenNebula Project

DDC Demo

OpenNebula Project

Cloud Disaggregation with OpenNebula

OpenNebula Project

Mehr von OpenNebula Project (20)