Client Deployment of IBM Cloud Private (Think 2019 Session 5964A)

IBM Confidential
Client Deployment of IBM Cloud Private
#5964A
—
Michael Elder
IBM Distinguished Engineer – IBM Multicloud
Platform
@mdelder
Yong Feng
IBM Senior Technical Staff Member – IBM
Cloud Private
@luckyfengyong
Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.

What’s
included in
IBM Cloud
Private? 3Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

Reference Architecture
4
http://ibm.biz/icpreferencearch

Topology Architecture
5
http://ibm.biz/icptopologyarch

Available Resources
6
http://bit.ly/icp-planning
Operator guides were produced as a joint
effort between engineering, support, and
teams in the field
Designed to provide real world guidance
Always under improvement – give us your
feedback!

How should
you plan
your specific
architectureThink 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation

8
Critical Architecture Decisions
HA Workload Feature
• How many failed nodes
can be tolerated?
• Do you need
availability zones?
• Upgrade with zero
downtime?
• What characteristics
define your workload:
cpu-intensive, memory-
intensive or others?
• What phase delivery
lifecycle: dev, test, UAT
or production?
• What is your required
throughput from your
consumers?
• Monitoring?
• Logging?
• Metering?
• Vulnerability Scan?
Security
• Do you need stringent
isolation for multiple
cluster consumers?
• Is certificate
management required?
• Is full PCI compliance
required?
• SELinux and firewall?

How should
you design
a cluster?

Provision
infrastructure
Prepare external
service
Ready for installation
1
# of
cluster
How many
clusters?
2
Host
group
What kind of host
group, how many
hosts in the host
groups and what’s
the size of hosts?
3
Network
Network
topology, ingress
of management
control plan and
user workload
4
Storage
Storage for
management
services and
user workload
Five Key Principles Define IBM’s Approach…
5
Infra
Infrastructure
utilities leveraged
from infrastructure
manager
Design cluster with six key factors
6
Config
Configuration of OS
of host,
configuration of
management
services,
configuration of
external services
Installation
configuration

# of Cluster
11
The overall approach within the enterprise
Cost:
Aligned with the organisational units
Network latency:
Aligned with the geography
Scalability:
Aligned with the size of the managed nodes
Environment Requirement:
Aligned with the number of the distinct environment such as test, UAT and
production (whether namespaces isolation achieves the desired goals?)

Host Group
12
Determine optional host group
• etcd: Enable for large scale cluster
• management: Enable when loads of management services are high
• va: Enable when vulnerability advisor and mutation advisor are enabled
• proxy: Enable when throughput of accessing to services from outside
cluster is high
Determine resource isolation
• Dedicate proxy node for a namespace
• Dedicate worker node for a namespace

Number and Size of Host
13Think 2019 / 5964A / Feb 15, 2019 / © 2019 IBM Corporation
Machine Role Number vCPU (>= 2.4 GHz) Memory Disk Space Comment
Master 3 16 32GB 500GB 3 for HA
Management 2 16 32GB 500GB 2 for HA
Proxy 2 4 16GB 400GB 2 for HA
Vulnerability Advisor 1 8 32GB 500GB Optional (none-HA)
Worker Nodes 2-50 8 32GB 400GB
A typical production environment
http://ibm.biz/icpcapacityplan

Management Node considerations
Separate management node from master node
– CPU, Memory and Disk intensive services running in management nodes
Increate the number of management node for large cluster
– Adding more management nodes not only increase the high availability but
balance the load of management services

Proxy Node considerations
Proxy nodes scale better
vertically vs horizontally as
shown in the figure. Notice
that 1 Proxy Node of 8
vCPU supports nearly the
same workload as 3 Proxy
Nodes of 4 vCPU
– Rather than adding more
nodes, it is better to increase
the size of the node

Network - Resources
External Load Balancer
– ELB for master node
– ELB for proxy node
VIP
– Recommend ELB for
Production environment
Container network
– Network policy
Host network
External Network Controller

Network - Firewall Rule
Protocol within cluster
– ipip (94) of IPV4
Port number
– Externally access to master and proxy nodes
– Internally access between master, proxy, management, va, etcd and worker
nodes
http://ibm.biz/icpportnumber

Network - DNS
DNS resolving of services
– <service>.<namespace>.sv
c.<cluster_domain>
Join upstream DNS chain
– Pick up upstream DNS
configuration from host
automatically
– Specify upstream DNS
configuration explicitly

Storage – Management Service
February 15, 2019 ICP Solutioning Guide 101 | IBM Confidential | IBM Cloud Solutioning Centers
Shared storage
– Image Registry: Large capacity which depends on the number of images
– License Audit Log: Small capacity
Local storage
– Docker: https://docs.docker.com/storage/storagedriver/select-storage-driver/
– etcd: High IOPS, SSD is preferred
– MongoDB: SATA is OK, but SSD is better.
– Elasticsearch: Large capacity

Storage – User Application
Storage options hosted on IBM Cloud Provider cluster
– GlusterFS
– Ceph block storage by using Rock
– Minio
Storage options hosted outside IBM Cloud Provider cluster
– vSphere storage provider
– Network file system
– IBM Spectrum Scale
Storage options allowed by Kubernetes
– https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes

Storage - Backup
Kubernetes cluster state
– etcd: http://ibm.biz/icpbackup
Persistent volumes
– Traditional base backup tools can be used for backing up nodes and file system.

Storage – Backup with VM Solution
February 15, 2019
http://ibm.biz/icpbackupwithvmware

Infrastructure Provider
Infrastructure Metadata
– Host topology such as available zone
– Labels
Network
– NSX-T by vSphere
– ALB/ELB by AWS
– F5
Storage
– Datestore by vSphere
– EBS by AWS

Infrastructure Provider (Cont’d)
AWS
– AWS Cloud Provider
https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws
vSphere
– vSphere Cloud Provider
https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/overview.html
F5
– F5 Network Solution
https://clouddocs.f5.com/containers/v2/kubernetes/

Configuration - OS
Security
– Enable Selinux to get better security protection on host resource
– Enable local firewall to get better security protection on network
Kernel Parameter
– Network related parameters
http://ibm.biz/icposkernelparam
– Virtual memory related parameter for elasticsearch
vm.max_map_count

Configuration – External Service
LDAP/AD
– Prepare LDAP/AD for user authentication
http://ibm.biz/icpldap
Key Management Service
– Prepare KMS for secret encryption
http://ibm.biz/icpkmssecret
Vault
– Prepare HashiCorp Vault for certificate manager
http://ibm.biz/icpvault

Configuration – Management Service
Docker
– Follow CIS security benchmark
– Storage driver
Kubernetes
– Scheduler policy
– Configuration for large cluster: http://ibm.biz/icplargecluster
etcd
https://coreos.com/etcd/docs/latest/tuning.html
ELK
http://ibm.biz/icpelktuning

Examples

IBM Cloud Private in
AWSLeverage available zone
– Master/mgmt/va across available
zone
– User application across available
zone
AWS ALB/NLB
– Load balancer for management
control plane
– Load balancer for user application
– Security group to control network
access
EBS as persistent storage
http://ibm.biz/icponaws

Large Scale Cluster (1000 nodes)
Size of host
– etcd/Master/Management/Proxy/VA: 36 CPU, 60 GM memory and 10 GB networ
OS kernel parameter
– Network and virtual memory: net.core.somaxconn, net.ipv4.neigh.default.gc_thresh, fs.file-max …
Calico
– Enable router reflector
etcd
– --heartbeat-interval=500, --election-timeout=2500, --snapshot-count=5000
Kubernetes
– memory cache, communication timeout, API throttle, parallelism of ops

Multiple Tenants with Isolation
Proxy in DMZ can only
access service from
tenant A
Proxy in intranet can only
access service from
tenant B
Services from tenant A
and Service from tenant B
are running in different
workers and cannot
access between each
other

Air-gapped Environment
Proxy configuration for Docker
## Docker environment setup
docker_env:
- HTTP_PROXY=http://1.2.3.4:3128
- HTTPS_PROXY=http://1.2.3.4:3128
- NO_PROXY=localhost,127.0.0.1,{{ cluster_CA_domain }}
Proxy configuration for helm-api
tiller_http_proxy: http://1.2.3.4:3128
tiller_https_proxy: http://1.2.3.4:3128
http://ibm.biz/icpairgapped

Notices and disclaimers
© 2019 International Business Machines Corporation. No part of this
document may be reproduced or transmitted in any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity. IBM products and
services are warranted per the terms and conditions of the agreements
under which they are provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are presented as
illustrations of how those
customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
References in this document to IBM products, programs, or services does
not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared
by independent session speakers, and do not necessarily reflect the views
of IBM. All materials and discussions are provided for informational
purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal
requirements and to obtain advice of competent legal counsel as to
the identification and interpretation of any relevant laws and regulatory
requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not
provide legal advice or represent or warrant that its services or products
will ensure that the customer follows any law.

Notices and disclaimers
continued
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products about this
publication and cannot confirm the accuracy of performance, compatibility
or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of
those products. IBM does not warrant the quality of any third-party
products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM expressly disclaims all
warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM
products and services used in the presentation] are trademarks of
International Business Machines Corporation, registered in many
jurisdictions worldwide. Other product and service names might
be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at “Copyright and trademark
information” at: www.ibm.com/legal/copytrade.shtml.

IBM Confidential
Thank you

®
https://www.ibm.com/legal/us/en/copytrade.shtml

Client Deployment of IBM Cloud Private (Think 2019 Session 5964A)

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Client Deployment of IBM Cloud Private (Think 2019 Session 5964A)

Ähnlich wie Client Deployment of IBM Cloud Private (Think 2019 Session 5964A) (20)

Mehr von Yong Feng

Mehr von Yong Feng (10)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Client Deployment of IBM Cloud Private (Think 2019 Session 5964A)

Hinweis der Redaktion