What does it really take to make sure your application is production-ready? With new privacy regulations being added, many aspects need to be taken into account when deciding when to deliver your final product to the public. Can your application handle multiple users with different levels of access? Can you extend your application to use existing authentication and authorization platforms? Have you invested in using Mutual TLS authentication for communication between components? How do you manage the certificates and passwords used within your application? Does it connect to the database securely? This talk will cover all aspects from start to finish on how to deliver production-ready code.

1. Creating Production-Ready,Secure and
ScalableApplicationsin IBM Cloud Private
(#4789A)
—
Shikha Srivastava
IBM Senior Technical Staff Member
@shikhasthoughts
Michael Elder
IBM Distinguished Engineer - IBM Multicloud Platform
@mdelder
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation

2. Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future productsis intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future productsis not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for
our productsremains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks
in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.
2

3. IBM Confidential
Notices and disclaimers
3Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation
© 2019 International BusinessMachinesCorporation. No part of this
document maybe reproducedor transmittedin any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use,duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations(including information relating to
products that have not yet been announcedby IBM) hasbeen reviewed
for accuracyasof the date of initial publication andcouldinclude
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied.In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity.IBM products and
servicesare warrantedper the termsand conditionsof the agreements
under which they are provided.
IBM productsare manufacturedfrom newpartsor new and used parts.
In some cases, a product may not be new and mayhave been previously
installed. Regardless, our warrantytermsapply.”
Any statements regarding IBM's future direction,intent or product
plans are subject to change orwithdrawal without notice.
Performance data containedherein wasgenerallyobtainedin a
controlled, isolated environments. Customer examplesare presented as
illustrationsof how those
customershave used IBM productsand the results they mayhave
achieved. Actual performance, cost, savingsor other results in other
operating environmentsmay vary.
Referencesin thisdocument to IBM products, programs, or servicesdoes
not implythat IBM intendsto make such products, programsor services
available in all countriesin which IBM operates or does business.
Workshops, sessions and associatedmaterialsmayhave been prepared
by independent session speakers, and do not necessarily reflect the
views of IBM. All materialsanddiscussions are provided for
informational purposesonly, andare neither intendedto, nor shall
constitute legal or other guidance or advice to any individual participant
or their specific situation.
It isthe customer’sresponsibility to insure itsown compliance with legal
requirementsand to obtain advice of competent legal counsel asto
the identification andinterpretation of anyrelevant laws and regulatory
requirementsthat mayaffect the customer’sbusiness andany actions
the customer mayneed to take to complywith such laws. IBM does not
provide legal advice or represent or warrant that its servicesor products
will ensure that the customer followsany law.

4. IBM Confidential
Notices and disclaimers
continued
4
Information concerning non-IBM products was obtainedfrom the
suppliers of those products, their published announcementsor other
publicly available sources. IBM has not tested those products about this
publication andcannot confirm the accuracyof performance,
compatibilityor anyother claimsrelatedto non-IBM
products. Questionson the capabilities of non-IBM products should be
addressed to the suppliers of those products. IBM does not warrant the
quality of any third-party products, or the abilityof any such third-party
products to interoperate with IBM’s products. IBM expressly disclaims
all warranties, expressed orimplied, including but not limited to, the
implied warranties of merchantability and fitness fora purpose.
The provision of the information containedherein isnot intendedto, and
does not, grant anyright or license under any IBM patents, copyrights,
trademarksor other intellectual propertyright.
IBM, the IBM logo, ibm.com and[names of other referencedIBM
products andservices used in the presentation]are trademarksof
International Business MachinesCorporation, registeredin many
jurisdictionsworldwide. Other product and service namesmight
be trademarksof IBM or other companies. A current list of IBM
trademarksisavailable on the Web at “Copyright and trademark
information” at: www.ibm.com/legal/copytrade.shtml.
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation

5. What is Ready for production application
• Secure
• Installation, authentication and access
• Resilient, HighlyAvailable and scale
• Repeated deployment
• with safe upgrades and configuration
changes
• Performance
• Observable
• Upgradeable
• more …..
• And AGILE too
Its not MAGIC J

6. 6
What is a
12-factor
app?
https://12factor.net/
• “12-Factor” is a software
methodology for building scalable
microserviceapplications
• Originallycreatedby Heroku
• Best practicesdesignedto enable
applicationsto be built with
portability,resilience, and
scalability when deployed to the
web

7. 7
I. Codebase
One codebase tracked in revision control, many deploys
II. Dependencies
Explicitlydeclareand isolate dependencies
III. Config
Store config in the environment
IV. Backingservices
Treat backing services as attached resources
V. Build, release, run
Strictlyseparatebuild and run stages
VI. Processes
Executethe app as one or more stateless processes
VII. Port binding
Export services via port binding
VIII. Concurrency
Scaleout via the process model
IX. Disposability
Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity
Keep development, staging, and production as similar as
possible
XI. Logs
Treat logs as event streams
XII. Admin processes
Run admin/management tasks as one-off processes
Why
12 factor
apps?
• Make it easier to run, scale, and
deploy applications
• Keep parity between development
and production
• Provide strict separation between
build, release, and run stages

8. I. Codebase
One codebase tracked in
revision control, many deploys
II. Dependencies
Explicitly declare and isolate
dependencies
III. Config
Store config in the environment
IV. Backing services
Treat backing services as
attached resources
V. Build, release, run
Strictly separate build and run
stages
VI. Processes
Execute the app as one or more
stateless processes
VII. Port binding
Export services via port binding
VIII. Concurrency
Scale out via the process model
IX. Disposability
Maximize robustness with fast
startup and graceful shutdown
X. Parity between dev & prod
Keep development, staging,
and production as similar as
possible
XI. Logs
Treat logs as event streams
XII. Admin processes
Run admin/management tasks
as one-off processes
Code Deploy Operate

9. Developers dream – Code factors
9
• One codebase for my application
tracked in revision that runs
anywhere: build, ship and run
anywhere
AND
• I can offload deployment, HA,
scaling, upgrade strategy and not
worry about it
Test and
automation
Release AGILE
Design
• Container Images built from
Dockerfiles using trusted small image.
Kubernetes Deployments, etc
managed as YAML (F#I- Codebase)
• Having a strong artifact-driven model
makes it easier to follow a
Continuous Delivery lifecycle (F#V-
Build, release, run)
• Using the same images and YAML
objects make it easier for dev teams
to match what’s running in
production
(F#X- Dev/prod parity)
Develop

10. Pod
(Single IP Address)
Deploy factors
10
• ConfigMaps and Secrets managed in source
repositories or built dynamically via commands
(F#III: Config ). Containers retrieve during runtime
• A collection of Pods can expose or consume
Services via Service port bindings (F#IV : Backing
Services, F#VII: Port binding)
• Container image runs as a container process in a
Pod with other containers (F#VI: Processes )
• Explicitly declare and isolate dependencies(F#II:
Dependencies)
• Running app as a container makes it possible to
capture all logs, metrics, and other management
functions in a consistent way (F#XII: Admin
Process )
Pod
(Single IP Address)
Volume
Volume
container
container
container
Volume
Secret
ConfigMap
Service
Persistent
Volume

11. • Ensure scale for your app
• Replica set ensures specified number of pods are always
running
Service
Replica set
container
Deployment
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
template:
metadata:
labels:
service: http-server
spec:
containers:
- name: nginx
image: nginx:1.10.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
• Is this enough?
Remember load is never constant in the real world
container
Operate factors: Concurrency (F#VIII) & Disposability (F#IX)

12. Operate factors: Concurrency (F#VIII)
Leverage autoscaling to automate computation resources based on load
• Horizontal Pod Scaler (HPA)
• Controls the number of replicas
• Use cpu or memory as a trigger or use
custom metric
• Applicable for stateless app
• Vertical Pod Scaler (HPA)
• Controls the memory and cpu for pod
• Use cpu or memory as a trigger or use
custom metric
• Applicable for statefull apps
Service
Replica set
Pod
Deployment
Pod Pod---
HPA
(policy)
Resource
metric
(usage)
Service
Replica set
Deployment
VPA
(policy)
Resource
metric
(usage)
Pod
Pod
Pod
Pod

13. 13
7
missing
factors
XIII. Observable
Apps should provide visibility about current health and metrics
XIV. Schedulable
Apps should provide guidance on expected resource constraints
XV. Upgradable
Apps must upgradedata formats from prior generations
XVI. Least privileged
Apps should provide guidance on expected resource constraints
XVII. Auditable
Apps should provide appropriateaudit logs for compliance needs
XVIII. Access Control (Identity, Network, Scope, Certificates)
Protect app and resources from the world
XIX. Measurable
Apps usage should be measurable for quota or chargebacks

14. IBM Cloud / © 2018 IBM Corporation
Observable: Application health (F#XIII)
Know your applicationhealth
• Kubernetesprobes
• Is the app ready to accept
traffic?:Readiness
• Is the app responsive?:
Liveliness
• Is this enough?
• What about transactions,
traffic, memoryusage ?
App instance 1
App instance 2
Requests
/metric
/metric
livenessProbe:
# an http probe
httpGet:
path: /healthcheck
port: 8080
initialDelaySeconds: 15
timeoutSeconds: 1
readinessProbe:
# an http probe
httpGet:
path: /readiness
port: 8080
initialDelaySeconds:
20
periodSeconds: 5

15. ClusterIBM Cloud / © 2018 IBM Corporation
Schedulable: Resource requests, limits, & quotas (F#XIV)
• Guarantee resources for your containers: Specify request and limits for the compute resources
CPU request: 150 CPU limit: 200
Guaranteed
CPU
Throttle limit
for K8
Max CPU resource for
container
CPU request: 0 CPU limit:0
Max CPU resource for
container
No request and limits are set. It defaults to 0
No guarantees, pods can be preempted any time
Once quota in a namespace for compute resources set, the users are forced to set requests or limits for
those values
• Set resource quota
Namespace 1
Resource Quota :
CPU Limit:500mi
Memory Limit: 1024
MIB
Namespace 2
Resource Quota:
CPU Limit:500mi
Memory Limit: 1024
MIB

16. IBM Cloud / © 2018 IBM Corporation
Upgradable (F#XV)
Applications should be able to roll out updates for cases where
backward compatible updates ( security or feature updates )needs
to be made
Servicecontainer
Deployment
Replica set V2Replica set V1
containercontainer container container container
minReadySeconds: 5
strategy:
# indicate which strategy
# we want for rolling update
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1

17. Cluster
IBM Cloud / © 2018 IBM Corporation
Least Privilege(F#XVI)
• Limit container access to hosts
• Every permission is an attack vector
• Use Pod Security Policy and Network Policy
to
• Limit access to filesystem
• Limit access to Kernel capabilities
• Use a non-privileged user
• Limit access to volume types
• Limit access to ports
container
container
container
container
#sample-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: false
# Don't allow
# privileged pods!
# The rest fills in some
# required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
volumes:
- '*'
Compromised

18. IBM Cloud / © 2018 IBM Corporation
Auditable (F#XVII)
• Know WHAT/WHEN/WHO/WHERE for all CRUD operations
• Chronological set of records documenting sequence of events affecting
system and application by users or components
• Use cloud agnostic industry standard format – CADF (Cloud Auditing Data
Federation)
• Control the quantity of logs
CADF event:
<initiator_id>: ID of the user that performed the operation
<target_uri>: CADF specific target URI, (for example: data/security/project)
<action>: The action being performed, typically: <operation>. <resource_type>

19. IBM Cloud / © 2018 IBM Corporation
Access Control -Identity, Network, Scope (F#XVIII )
• Protect app and resources from the world
• Authentication and Authorization
• Certificate Management
• Data Protection
• Network security
• Network policy
• Network Isolation
• Admission Controller
• Example: Image admission controller

20. Access Control: Identity, Network, Scope (F#XVIII)
• Ensure secure communication
• Generate Certificates
• Enable TLS / mTLS
• Manage Certificates
letsencrypt-stagingletsencrypt-prod icp-root-ca
signed
keypair
signed
keypair
Certificate Manager
signed
keypair
k8sSecret
Issuer
Certificates
Secrets
1. Issuer createsCertificate
2. Certificate createssecret
3. Secret mountsto
Pod
IP: 9.37.239.158
Issuer: icp-root-ca
IP: 9.37.239.158
Issuer: icp-root-ca
Example.com
Issuer: letsencrypt-prod
# sample issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: demo1-nginx-ca
namespace: demo
spec:
ca:
secretName: demo1-nginx-ca-key-pair
# sample certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
Metadata:
name: demo1-nginx-cert
spec:
secretName: demo1-nginx-cert
issuerRef:
name: demo1-nginx-ca
kind: Issuer
commonName: "foo1.bar
dnsNames:
foo1.bar1
Pod

21. Cluster container
container
container
container
container
container
---
IBM Cloud / © 2018 IBM Corporation
Measurable (F#XIX)
Dept/org 1 Dept /orgn
Total usage
Cluster containercontainercontainercontainer containercontainer
IT cost
Dept /org 2 ---
---
• Know the cost of the application
• Compute resources allocated to run the containers should be measurable
• Org / department using the cluster should be accountable

22. So, What really
makes a
production-
ready app? 22

23. IBM Cloud / © 2018 IBM Corporation
A production grade application
Production
thinking needs
to be through
the entire
process
Attention to
Building containers and what's inside the containers
Example: Factor I : codebase , Factor X: dev/prod
parity, Factor XV
Attention to
Kubernetes configuration
Example: Factor III: Config, Factor II Config, Factor
XIV: Schedulable
Attention to
Cloud provider configurations
ExampleXII: Observable, Example: XVIII: Access
Control. Factor XIX:: Measurable

24. IBM Cloud / © 2018 IBM Corporation
IBM Cloud Private (ICP)
24
• Self-service rich catalog of IBM MW
• Helm based parameterized install to
simplify complex K8 apps
• Logging : ELK + filebeat
• Monitoring : Prometheus + Grafana
• Usage : IBM Metering Service
• IBM Vulnerability Advisor
• IBM MutationAdvisor
• Authentication/ Authorization
• Certificate Management
• Network security
• Audit trail for any CRUD operations
• Team based organization of resources
Enterprise Content Catalog
Open SourceandIBM Middleware, DevOps,
Data, Analytics, and AI Software
Core Operational Services
Logging,Monitoring,Metering, Security,Alerting
Kubernetes Container
Orchestration Platform
IBM Z
Choice of
infrastructure:
All communication enabled over TLS.
Data secured in transit and at rest
Provides the capabilities to run containerized application in secure, scalable and resilient
environment

25. Enough
talking,
let’s see it
LIVE! 25

26. Leverage the IBM Cloud Garage
Method to change how you work.
26Think 2018
Provides an in-depthcollection of
practices,tutorials,and
architecturesto help you on this
journey.
Completelyopen forum for learning
at your own pace.
We offer hands-on guidance and
services,if needed.
Defined
Practices
Business
Benefits
Technical
Benefits
ibm.com/cloud/garage>

27. 27
Learn
more in
our new
book!
#7678A: Tech Talk:
Deploying Kubernetes in
the Enterprise (with the
authors)
When: Wednesday, 11:30
AM - 12:10 PM
Where: Table Top Tap Room
at the Metreon | Code Cafe
Tech Talks Area
Get a signed copy with all
of the authors at the Code
Café Mezzaine on
Wednesday (7 – 7:30PM)!
ibm.biz/BdYA4i>Now available online compliments of IBM:

28. Read the white paper!
28
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM
Corporation
http://ibm.biz/Ovum-WP
84% improvement in speed to market
75% reduction in operational expenses
60% increase in projects IT
Administrators can handle
50% reduction in service desk calls

29. 29
®
https://www.ibm.com/legal/us/en/c opytrade.s html