Introducing an internal cloud brings new paradigms, tools and infrastructure management. When placed alongside traditional HPC the new opportunities are significant But getting to the new world with micro-services, autoscaling and autodialing is a journey that cannot be achieved in a single step.

1. Flexible Compute Environment
•We have successfully been using the Farms for a number of
years.
•These are a fantastic tool for our science.
•But now - new cloud technologies are available.
•We want to be able to take advantage of these industry
standard technologies in our own private cloud – whilst still
retaining the benefits of the farms.

2. HPC and Cloud computing are
Complimentary
Traditional HPC
The highest possible
performance for Sanger
workloads
• A mature and centrally managed
compute platform.
• High performance Lustre filesystems
and system libraries optimised for
heavy data centric workloads.
• Efficient fair-share workload
management across large scale
infrastructure
• The ability to add bespoke or
disruptive technologies, such as
FPGA or GPU accelerators.
Flexible Compute
A collaborative platform,
with security and flexibility.
• Full segregation between projects
ensures data security throughout
computation tasks.
• Offers standard API’s and tools including
Docker to our developers and community
of collaborators.
• Scale-out computing with the ability to
over commit CPU allocation ensure
efficient resource utilisation
• Developers and collaborators are no
longer tied to a single operating system or
shared libraries. They are free to follow
the latest technologies and trends

3. Cloud computing: a flexible
platform for collaborative
research
•Cloud computing permits our customers to run their own images within a
secured network environment. This means that:
•We are no longer tied to the operating system and libraries that run on todays HPC
platforms.
•We can distribute our code in a pre-packed format that others can share.
• By sharing these images we can send Sanger pipelines to collaborators without technical
knowledge. This sends the workloads to where the data resides. (Not data to the work.)
•Cloud computing provides common API’s and libraries to our research community.
•We can support disruptive technologies, including Docker in a secure environment,
accelerating development of new scientific pipelines and software providence.
•We can overcommit CPU usage by at least 1.5 : 1. This allows for greater resource efficiency
and cost management.

4. Cloud computing collaboration
across sites
Open Standard
APIs
Federated
AAAI
MRC Bioinformatics
Centres
Public
clouds
WTSI Flexible
Compute
Common Collaborative API’s,
and familiar self service
environments that are secure
by design

5. Future HPC and cloud
computing integration.
OpenStack
CASM
HPC cluster
Cloud
environment
Humgen
Core …
Software Defined Networking
(Infrastructure as Code, IaaS)
It is becoming possible to install and manage
traditional HPC infrastructure using the same
tools that are provided by our cloud
environment
This would allow us to:
• Provide a consistent securable network
infrastructure across all computation
resources.
• The provision of both HPC resources in a
traditional manner and access to cloud
resources on a per tenant basis.
• Provide the best of both worlds for our
customers.
• Reduce management overheads
• Expand to meet future demand
• Accommodate disruptive technologies as
they arise.
Build from a menu requires
no user knowledge….

6. Sanger flexible compute
platform is coming soon
•Our new flexible compute platform has been under acceptance and early testing for over
a month.
•We’ve had excellent feedback from our research customers.
•On release, this platform will provide:
• 9000 vCPUs
• > 50Tb memory
• 1Pb of storage, including externally accessible S3 object storage.
Released on Wednesday the 22nd of February 2017
For more details, please contact Peter Clapham or Tim Cutts.

7. OpenStack day on Campus
•On March the 10th we are hosting an OpenStack day
•Guest speakers from the wider community.
•We have Tim Bell from CERN as a key speaker.
•Tim is responsible for CERN’S Openstack environment of > 200,000 cores.
•Other attendees include representatives from:

8. Production openstack (I)
• 107 Compute nodes (Supermicro) each with:
• 512GB of RAM, 2 * 25GB/s network interfaces,
• 1 * 960GB local SSD, 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz )
• 6 Control nodes (Supermicro) allow 2 openstack instances.
• 256 GB RAM, 2 * 100 GB/s network interfaces,
• 1 * 120 GB local SSD, 1 * Intel P3600 NVMe ( /var )
• 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz )
• Total of 53 TB of RAM, 2996 cores, 5992 with hyperthreading.
• Redhat Liberty deployed with Triple-O

9. Production openstack (II)
• 9 Storage nodes (Supermicro) each with:
• 512GB of RAM,
• 2 * 100GB/s network interfaces,
• 60 * 6TB SAS discs, 2 system SSD.
• 2 * Intel E52690v4 ( 14 cores @ 2.6Ghz )
• 4TB of Intel P3600 NVMe used for journal.
• Ubuntu Xenial.
• 3 PB of disc space, 1PB usable.
• Single instance ( 1.3 GBytes/sec write, 200 MBytes/sec read )
• Ceph benchmarks imply 7 GBytes/second

10. Production openstack (III)
• 3 Racks of equipment, 24 KW load per rack.
• 10 Arista 7060CX-32S switches .
• 1U, 32 * 100Gb/s -> 128 * 25Gb/s .
• Hardware VxLan support integrated with openstack *.
• Layer two traffic limited to rack, VxLan used inter-rack.
• Layer three between racks and interconnect to legacy systems.
• All network switch software can be upgraded without disruption.
• True linux systems.
• 400 Gb/s from racks to spine, 160 Gb/s from spine to legacy
systems.
(* VxLan in ml2 plugin not used in first iteration)

11. But what are we providing?
CloudForms
Service driven access
OpenStack Horizon
Granular control over
instance
Direct API Access
Direct https access
from anywhere
Accessible only from
within Sanger
Ceph Object Storage
(Used to provide volume
and image storage)
S3 Object
Storage
Layer

12. How Does this Fit with Existing
Services?
OpenStack “Bubble”
Compute
Ceph
100GB/s SDN network
infrastructure
Sanger internal systems
Access to secured
services
i.e.
iRODS
Databases
CIFS (Windows shares)
S3 API access
OpenStack API and GUI
access
80GB/s connectivity
No access to:
NFS
Lustre

13. CloudForms Interface

14. Horizon Interface

15. Efficient Resource
Management
OpenStack resources are managed at a tenant group level.
• Each “tenant” group has an assigned quota for:
• Disk
• CPU
• Memory
Once limits are full, tenant members will either have to wait for resources to
become available or shutdown or terminate a running instance.
Initial quotas are agreed with the IC before creation

16. Quotas, they are not all the
same.
Some groups have a requirement that they have an absolute number of spots
available for essential services
Other groups would like to burst to meet demand as required.
These requirements do not fit well with each other.

17. The Proposed Workaround
For those projects which require guaranteed access:
• We create a dedicated tenant group that has specific access to a set quota
allocation of vCPU, Disk and memory.
• This is tied directly against reserved hardware
This guarantees requested resource will be available when required, whilst
providing security, operating system flexibility and instance management.
BUT there is no ability to use more than the requested allocation

18. Dynamic Workflows
Dynamic workflows can expand to meet demand and collapse when not required.
So a quota that matches the initial resource request will mean constantly under
quota’ing the system
For the initial release we will start by:
• Overcommitting CPU by 1.5 : 1 (available total vCPU ~9000)
• Overallocate quotas so that 115% of the overcommitted vCPU is available to
tenants.
So some initial ability to use more of the system than may be available.

19. For More Details, see
https://docs.google.com/a/sanger.ac.uk/document/d/17z9urhh3bTLRhQo9b8Ccs
ZW_3O7cxlGY9uiwpAS_GqQ/edit?usp=sharing
Or
http://tinyurl.com/zzurp5s
We are adding monitoring and metrics gathering to the system. This
will provide a feedback loop for quota and project management.

20. New Opportunities for
Application Development
Cloud application development aims to scale out compute and
provide:
• Auto scaling of key services
• Making pipelines cost effective on commercial platform providers
• Self-healing of service components that fail
• Creating resilient services with reduced impact when service components fail
• Not tied to any one specific environment
• Enabling sharing of code, images and services with collaborators. This can
dramatically reduce the need to copy large data sets around the world and permit
running complex pipelines where the data resides.

21. How do we see Migration ?

22. Initial Early Adopters.
We have some early adopters !
1. Mutational signatures
2. Imputation service
3. Blast service
4. Pan-prostate
We look forward to hearing more from
these groups soon !

23. Mostly Share a Common
Approach
Web
Interface
Data
upload
Run
analysis
Update job
status data base
Present data
Invoke
Analysis
Retain a copy

24. Adaption to Cloud based tools
Stage Current approach Cloud approach
User details local databases, directory services, Oauth Oauth, directory services
Data Downloads Globus or https S3, Globus
Job status RDBM: MySQL, Oracle or PostgreSQL NoSQL: Mongodb, Cassandra or REDIS
Invoke Job
Analysis
Hand crafted equest to LSF AMQP
Run Analysis LSF job submission AMQP, Heat orchestration or API call to Openstack
Present data Make available via sftp, Globus or https web
upload
S3 automatically generated URL's
Keep data No consistent approach S3, archive as required
Service failure Await systems Use IFTTT or add code to instance to raise or restart an instance as
required
Autoscale options Await systems Use IFTTT or add code to instance to raise or restart an instance as
required
Service discovery Manual Cloud init, heat templates, dynamic DNS