S104878 nvme-revolution-jburg-v1809b

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
IBM Systems Technical Events – ibm.com/training/events
2018 IBM Systems
Technical University
Sep 11-13, 2018
Johannesburg, South
Africa
Get Ready for the NVMe Revolution!
Tony Pearson
Master Inventor and Senior IT Architect,
IBM Corporation

Abstract
2
NVMe is the latest technological
breakthrough to drive lower latencies for all-
flash arrays.
This session will explain NVMe and NVMe
over Fabrics (NVMe-OF) and present IBM's
all-flash arrays that will support this
technology!

This week with Tony Pearson
3
Day Time Topic
Tuesday 11:30 Improve your NAS environment in One Day!
15:00
Dip your TOE in our Pool!
iSER and Data Reduction with Storage Virtualization
Wednesday 11:30
Save the World! Save your IT Budget with
IBM Cloud Object Storage System
15:00 How big is your NAS? Sizing, Management, and Deployment
16:15
Nightmares and Dreams:
Manage your entire Storage Infrastructure
Thursday 9:00
What else can you use that data for?
Adventures in Data Reuse
10:15 Be Persistent in your Journey to Private Cloud
13:45 Get ready for the NVMe Revolution

The Shifting Roles of Storage Technology
Flash and
Solid-State
Drives (SSD)
Combined with
slower Nearline
(7200rpm) disk
to reduce
energy costs
over faster
drives
“Flash & Stash”
Disk replication
and Virtual Tape
Libraries
Improved by low
cost Nearline
disk,
compression,
deduplication,
object protocols
Physical tape,
combined with
automation
Linear Tape File
System (LTFS)
Primary Data
Backup Data
Work Task
Project Folder
Long-term Data
Retention
4
Technology

How these Shifts are Happening
5
Storage arrays
designed for
spinning HDD
Storage arrays
designed for
Flash
Applications
Read and
Write files
Applications
GET and PUT
objects
NAS
Gateways
enable access
to Object store
and Clouds
Backup software
designed for
tape libraries
and drives
Virtual Tape
Libraries:
Disk pretends
to be tape
Backup software
designed for
Disk, Object
store, and
Clouds
Solid-state drives
(SSD):
Flash pretends
to be HDD
Compatibility Zone
NVMe
NVMeOF

Why NVMe for Enterprise Storage?
• New workloads and data access patterns require much higher storage performance
• Real-time big data analytics need an ability to support high degrees of concurrency
• Big data exacerbates the data mobility problem
With NVMe you can support workloads that have high IO requirements with
significantly smaller investment in infrastructure
6
Simplified
Management
Faster Time
to Results
Increased
Resource
Utilization
Enterprise
Solution

What is NVMe? (Non-Volatile Memory Express)
NVMe is a communications interface and protocol
• NVM Express defines an efficient interface for host software to communicate with a non-volatile
memory subsystem over PCI Express (NVMe over PCIe)
• Functionally analogous to SAS and SATA but designed to reduce overhead from drivers, OS and
application
NVMe uses PCIe fabric
• Multiple devices today in the market
• Multiple form factors including 2.5” drives
NVMe has been designed for high performance
• Increased IOPs, bandwidth and lower latency
• Exploit Flash and next-generation Non-Volatile
Memories
• Leverage multicore environments, high I/O
parallelism
7

Solid State Drives (SSD) vs. IBM MicroLatency™ Module
8
CUSTOM
Requires hardware
engineering
FAST
Designed for
low latency
RELIABLE
Innovative flash
management
COMMODITY
No hardware
skills needed
SLOW
Designed for
low cost
FALLIBLE
More single
points of failure
Better Performance Better Reliability Better Density Lower $/IOPS
IBM FlashCore™ Technology
Hardware Accelerated I/O
IBM MicroLatency™ Module
Advanced Flash Management
Spinning Disks
5-15 milliseconds
100-200 microseconds,
Up to 2 PB in 2U
~1 milliseconds
(1000 microseconds)
Solid State Drives (SSD)

Introducing Non-Volatile Memory Express (NVMe)
9
NVM Express (NVMe™) is an open collection of standards and information to fully expose the
benefits of non-volatile memory in all types of computing environments from mobile to data center.
NVMe™ is designed from the ground up to deliver high bandwidth and low latency storage access for
current and future NVM technologies
The register interface and command set for PCI Express attached storage with industry standard software available for
numerous operating systems. NVMe™ is widely considered the defacto industry standard for PCIe SSDs.
NVM Express Base Specification
http://nvmexpress.org/resources/specifications/
The command set and architecture for out of band management of NVM Express storage (i.e., discovering, monitoring, and
updating NVMe™ devices using a BMC).
NVM Express Management Interface (NVMe-MI™) Specification
The extension to NVM Express that enables tunneling the NVM Express command set over additional transports beyond PCIe.
NVMe over Fabrics™ extends the benefits of efficient storage architecture at scale in the world’s largest data centers by
allowing the same protocol to extend over various networked interfaces.
NVM Express Over Fabrics (NVMe-oF™) Specification

The Evolution of NVMe?
10
2014
2011
• NVM Express Spec 1.0
published by industry
leaders on March 1
2012
• NVMe Specification 1.1
released on Oct 11
• 2014: NVM Express Spec 1.2
released on November 3
• NVM Express Work Group
was incorporated at NVMe
Inc., the consortium respon-
sible for the development of
the NVM Express specification
• Work on the NVM Express over Fabrics
(NVMe-oF™) Specification kicked-off
• NVM Express Management Interface (NVMe-MI™)
Specification officially released. Provides out-of-band
management for NVMe™ components and systems and
a common baseline mgmt.
feature set across all
NVMe™ devices and
systems. 2015
• NVM Express over Fabrics (NVMe-oF™)
Specification published; extending
NVMe™ onto fabrics such as Ethernet,
Fibre Channel and InfiniBand®, providing
access to individual
NVMe™ devices &
storage systems.
2016
2018
IBM Delivers NVMe
& NVMe-oF
Solutions
2013
• 2013
Nothing
Happened
2017
• NVMe Spec
1.3 published.
Addresses the
needs of mobile
devices, with
their need for
low power and
other technical
features, making
it the only storage
interface available
for all platforms
from mobile
devices through
data center
storage systems.
1st Partial NVMe
solutions from
Competitors

NVMe – An Alternative to SCSI command set
11
Server
(SCSI Initiator)
Internal
flash and
disk media
External Device
(SCSI Target)
External
flash and
disk media
Server Internal
NVMe flash
External Device
(NVMe Storage
System)
External
NVMe
flash
CapsulePacket
Volume =
Logical Unit Number (LUN)
Volume =
Namespace
Identifier (NSID)
Small Computer System
Interface (SCSI)
• 34 commands
• For both flash and disk
• Can be sent over network such
as FCP, iSCSI, FCoE
• Terminology: LUN, Packet
Non-Volatile Memory
Express (NVMe)
• 15 commands
• For flash only
• Can be sent over network such
as FC-NVMe or NVMe-OF
• Terminology: NSID, Capsule

Does IBM need Non-Volatile Memory Express (NVMe)?
12
90 usec
Proprietary
PCiE
SCSI
FC, IB,
Ethernet
App
OS
Network
IBM FlashSystem 900
1000 usec
SCSI
SAS/SATA
SCSI
FC,
Ethernet
App
OS
Network
IBM SVC/Storwize
150 usec
NVMe
PCiE
NVMe
NVMe-OF
• RDMA
• FC-NVMe
App
OS
Network
Industry Trend
32Gb PCIe
faster than 12
Gb SAS
• 2x IOPS
for 70/30
workloads
• Lower
latency
Flash only
Streamlined
command set
• 64K
independent
queues
• 64K depth
• Less CPU

Evolution to exploiting full NVMe End to End Potential
13
Host
OS / Application
Connection Technologies
Media
All three
needed for
full
exploitation
Today’s Flash – Controller to Flash
Tomorrow’s SCM:
• PCM, MRAM, ReRAM, NRAM…
Memory Extension / DIMM (FlashCache)
NVMe over Fabrics (NVMe-oF)
• iWARP, RoCE, IB
• FC-NVMe
OS / Application user space
• Bypass the legacy stack
• Leverage the user-space I/O paradigm (with
Intel's SPDK, etc.)

End-To-End NVMe Technologies
14
Within Storage Controller
End-to-End NVMe Protocol
Media
Flash Drive
Host
OS/Application
User Space
Media
Flash Cache
Controller to Host
Connection
Technologies
FC-NVMe
FCP
NVMe-oF (RDMA)
iWARP / RoCE
Ethernet
NVMe-oF (RDMA)
InfiniBand NVMe protocol
For Flash Storage Connection
Three form factors:
o Add-in PCIe cards
o M.2 SSDs
o 2.5-inch U.2 SSDs
Persistent
Memory

Introducing the IBM FlashSystem 9100
15
Multicloud Solutions for Data Reuse, Business Continuity,
Private Cloud, and Storage Insights
Spectrum Virtualize (2 nodes per controller) with
AI-enabled software features
Highly scalable design (up to 8 nodes with 3,040 drives)
NVMe-accelerated performance, NVMe-OF Ready
Multiple drive options: FlashCore Modules, Industry Standard
NVMe, SAS-attached SSD expansion drawers
Backed by IBM FlashWatch guarantee programs

IBM FlashSystem 9100
FlashSystem 9110 and FlashSystem 9150
16
Dual Active-Active Array Controllers w/NVMe to Flash Media
Dual-ported 2.5” NVMe Flash bays (24)
Interface Card Slot (3) 10GbE Ports (4)Tech Port USB Ports (2)2KW PSU (2)
Flash Core Modules redesigned to 2.5” industry standard
form factor
(Includes: HW Compression, HW Encryption and 64 Layer 3DTLC)
FS9110
Dual 8-core
processors per
controller
FS9150
Dual 14-core
processors per
controller
Up to 1.5TB
Cache per
controller
enclosure
Up to 3 host
adapters per
controller

IBM FlashCore Module
IBM Engineered
Massively Parallel Design
FPGAs in the Data Path
Distributed RAM
High Speed Interface
Hardware-based Data
at Rest Encryption
and Embedded compression
Small Medium Large
AE3 card 3.6 TB usable 8.5 TB usable 18 TB usable
FCM w/NVMe 4.8 TB usable 9.6 TB usable 19.2 TB usable
17
FCM w/NVMe
AE3 card

FlashSystem 9100 with Spectrum Virtualize
18
Data Footprint Reduction
• Fully-allocated or Thin Provisioned volumes
• Data Deduplication
• Multiple compression options
Easy Tier for sub-LUN automated tiering
Data-at-Rest Encryption
• Multiple encryption options
Copy Services – FlashCopy, Remote Mirror, HyperSwap
FlashCore™ Embedded Hardware-Accelerated
Up to 3x, no performance impact Up to 5x, using Intel QuickAssist
co-processor
FlashCore™
Embedded
Hardware-
Accelerated
CPU-based
No performance
impact
For Industry NVMe,
and SSD expansions
For externally virtualized
storage

FlashSystem 9100 Storage Capacity
19
Drive Size 4.8TB 9.6TB 19.2TB 1.92 3.84 7.68 15.36
Array Usable 101TB 202TB 403TB 40TB 81TB 161TB 323TB
Array Effective 2:1* 202TB 403TB 806TB 81TB 161TB 323TB 645TB
Array Effective 5:1** 504TB 1008TB 2016TB 202TB 403TB 806TB 1613TB
This shows the USABLE TB (i.e. after DRAID 6 and a spare drive) and average EFFECTIVE TB (i.e. using 2:1 compression) capacities
in a single 24 drive enclosure. Do not size a system to exceed the MAXIMUM TB capacity
• Effective capacity assumes a 2:1 reduction in data, using inline hardware compression on 4.8TB and 9.6TB IBM NVMe FCM and DRP on the other drives
• ** Effective capacity assumes a 5:1 reduction in data using Data Reduction Pools (DRP) within the IBM Spectrum Virtualize software
IBM NVMe FlashCoreTM Modules Industry NVMe Drives

FlashSystem 9100 Upgrade Options
20
Start with 1 Control Enclosure
Add up to 40U Expansion Enclosures
24-By Controller
2.5-inch (SFF)
• 4.8TB, 9.6TB, 19.2TB FlashCore
• 1.92, 3.84, 7.68 and 15.36 TB NVMe
92-Bay in 5U expansion
2.5-inch (SFF)
• 1.92, 3.84, 7.68 and 15.36 TB SSD
Each can have up to 40U
Expansion Enclosures
24-Bay in 2U expansion
2.5-inch (SFF)
• 1.92, 3.84, 7.68 and 15.36 TB SSD
Up to 2.5M IOPS
34 GB/sec
760 drives
Cluster up to 4 Control Enclosures together into a single system
Up to 5M IOPS
68 GB/sec
1,520 drives
Up to 7.5M IOPS
102 GB/sec
2,280 drives
Up to 10M IOPS
136 GB/sec
3,040 drives

Interoperability with Existing Platforms
21
Storwize V7000
Storwize V7000F
(Gen 2 and Gen2+)
FlashSystem V9000SAN Volume
Controller
FlashSystem 9100
FlashSystem 9100
FlashSystem 9100
External
Virtualization External
Virtualization &
Data Migration
Up to 4
controllers can
be clustered
together
45 days of external virtualization
included no charge to assist with
data migration.

IBM Storage Utility Offering
22
* Assumes 42% average capacity utilization based on Industry data
By 2020, consumption-based procurement in datacenters will have
eclipsed traditional procurement through improved
"as a service" models
--- IDC FutureScape: Worldwide Datacenter 2018 Predictions
IBM Storage Utility Offering provides business the ability to pay for storage
capacity based on actual monthly usage
• Ready capacity when business dictates
• Pay for what’s used, when used
• Flex up and down monthly costs
• Powerful management capabilities of Storage
Insight for predictability and forward planning
• Available for FlashSystem 900 and new 9100
models, Storwize family, Cloud Object
Storage, and DS8000

How SCSI commands get sent from Servers to Storage Devices
23
Fibre Channel (FC)
Fibre Channel
Protocol (FCP)
Ethernet
Small Computer System Interface (SCSI)
Transmission Control Protocol (TCP)
Internet Protocol (IP).TCP/IP
Internet SCSI (iSCSI)
Host Bus
Adapter (HBA)
Network Interface
Card (NIC)
TCP Offload
Engine (TOE)
Fibre Channel is DETERMINISTIC
Flow control at source,
using buffer credits
Ethernet is NON-DETERMINISTIC
Flow control at destination
dropping packets, re-transmit

Data Center Bridging with Converged Ethernet
24
Ethernet
TCP/IP
Internet SCSI (iSCSI)Fibre Channel over Ethernet
(FCoE) attempted to bring
deterministic behavior over
Ethernet Converged Ethernet was required, 10GbE or
higher, with enhancements for Data Center
Bridging (DCB) like Priority Flow Control
Converged Ethernet
FCoE
Converged Network
Adapter (CNA)

Remote Direct Memory Access over Ethernet
25
Ethernet
iSCSI
Converged Ethernet
RDMA-capable
Network Interface
Card (R-NIC) for
iWARP
RDMA
iSER
UDP/IP
RoCE v2
Remote Direct Memory Access
(RDMA) reduces CPU overhead
by eliminating unnecessary
copies of data buffers
iSCSI Extensions over RDMA (iSER) take
advantage of RDMA on InfiniBand or
Ethernet
Internet Wide Area RDMA
Protocol (iWARP) is an
iSER protocol that uses
standard TCP/IP, but can
benefit from TOE or R-NIC
adapters
iWARP
RDMA over Converged Ethernet
(RoCE) v2 is an iSER protocol that
uses UDP/IP over Converged Ethernet
TCP/IPRDMA-capable
Network Interface
Card (R-NIC) for
RoCE v2

What about FC-NVMe and NVMe-OF ?
26
Fibre Channel (FC)
Fibre Channel
Protocol (FCP)
Small Computer
System Interface (SCSI)
Ethernet
Internet
SCSI (iSCSI)
Converged Ethernet
RDMA
iSER
UDP/IP
RoCE v2 iWARP
TCP/IP
FCoE
FC-NVMe NVMe-OF
Non-Volatile Memory
Express (NVMe) is an
alternative to SCSI
commands for Flash
drives only!
16 Gbps Host Bus
Adapter (HBA) are
FC-NVMe ready!
25 GbE R-NIC are
NVMe-OF ready!

NVMe over Fibre Channel (FC–NVMe)
Like RDMA, Fibre Channel is also a zero-copy data
transfer protocol
• Currently both SCSI and FC-SB (FICON) use FCP for low
latency data transfers
Brocade Gen 5/6 is NVMe ready
• Seamlessly integrate Gen 6 with next-generation NVMe over
Fabrics without a rip and replace
• FC-NVMe will run on existing Gen 5 and Gen 6 fabrics (16 / 32
Gbit fabrics) with FOS 8.2+
• Servers using FC-NVMe need Gen 6 host bus adapters (HBAs)
Cisco official NVMe support and validation for all
16Gbps/32Gbps MDS9000 switches with NX-OS 8.1+
27
Brocade NVMe over Fibre Channel for Dummies

Exploiting Dual Protocol FCP and FC-NVMe
Dual-protocol infrastructure for easiest migration
• Deploy NVMe based arrays
• Leverage existing infrastructure
• Easily supported dual infrastructure
• How long will the transition take?
• Avoid risks
• Incremental Migration
• Applications dictate how individual volumes can be migrated
• Changes can be rolled back easily without disruption to
hardware or cabling.
• FCP and NVMe over FC can both leverage FC zoning
• Only SAN Zoning improves security
• Zoning restricts devices from accessing network areas that
should not be visiting!
• Discovery
• Emulex create drivers that leverage FCP for device discovery,
then check those devices for FC-NVMe traffic support
Existing 32Gb FC fabric
SCSI-on-FC
FC-NVMe
SCSI on FC Arrays FC NVMe Arrays
Servers
SCSI-on-FC
FC-NVMe GEN6 HBAs
DUAL Protocol SAN
arrays
Existing enterprise
Storage infrastructure

Gartner’s Current Timeline for the evolution of Enterprise NVMe and NVMe-oF
29
Source: Gartner (February 2018)

IBM Roadmap for NVMe-OF for IB, Ethernet and Fibre Channel
30
NVMe
NVMeOF (RDMA)
InfiniBand
App
OS
FC-NVMe
On Feb 20, 2018 IBM
announced NVMe-oF IB support
for FlashSystem 900 model AE3
NVMeOF (RDMA)
iWARP / RoCE
Ethernet
IBM intends to enhance to support
NVMe-oF (Ethernet, using 25GbE cards)
• FlashSystem V9000 model AC3
• SAN Volume Controller model SV1
• Storwize models 624 and AF6
• FlashSystem 9100
IBM intends to enhance to support FC-NVME
• FlashSystem 900 AE3
• FlashSystem V9000 AC3, SVC SV1, Storwize models 624, AF6,
and FlashSystem 9100 (using 16Gb FCP quad-port cards)
Statement
of Direction

Value of Moving to NVMe Protocol
As a result of the simplicity, parallelism and efficiently of NVMe, it delivers significant Performance
gains vs SCSI
Some metrics include:
• For 100% random reads, NVMe has 3x better IOPS than 12Gbps SAS
• For 70% random reads, NVMe has 2x better IOPS than 12Gbps SAS
• For 100% random writes, NVMe has 1.5x better IOPs than 12Gbps SAS
• For 100% sequential reads: NVMe has 2x higher throughput than 12Gbps SAS
• For 100% sequential writes: NVMe has 2.5x higher throughput than 12Gbps SAS
In addition to just IOPS and throughput, the efficiencies of command structure described above also
reduce CPU cycles by half, as well as
reduce latency by more than 200 microseconds compared to 12 Gbps SAS
31
Source: Fibre Channel Solutions Guide 2016

Spectrum Scale and NVMe
Flash Memory Summit (FMS) 2017 announced that
an NVMe over Fabrics demonstration that included
IBM’s Spectrum Scale was named the 2017
recipient for “Most Innovative Flash Memory
Business Application.” IBM is a winner in this
particular Best of Show category for the second
year in a row
Solution can deliver 15 million IOPS at a latency of
around seven microseconds
32

Flash Memory Summit Award for 2018:
Most Innovative Flash Memory Technology – IBM FlashSystem 9100
33
“Customers demand new levels of performance, data
reduction and ROI from their storage investments,”
said Jay Kramer, Chairman of the Awards Program and
President of Network Storage Advisors Inc. “We are proud
https://www.ibm.com/blogs/systems/flash-memory-summit-award-innovative-flash-memory-technology/
August 9, 2018 (https://flashmemorysummit.com/)
to recognize the IBM FlashSystem 9100,
which delivers unmatched performance with
100µs latency, 2.5M IOPS and 34GB/sec
bandwidth and up to 5:1 data reduction
through block-level deduplication, hardware-
based compression technology, SCSI
UNMAP support, pattern matching, and thin
provisioning for an impressive cost effective
storage solution.”

Redbook: IBM Storage and the NVM Express Revolution
Highlights
• The industry recognizes the need for standardized and efficient protocols
and interfaces that are optimized for low-latency, performance-dense
storage media (such as flash) and non-volatile memory technologies. This
paper provides the following highlights:
• The NVM Express (NVMe) technology enables storage accesses with low latency, high
efficiency, and high scalability
• The NVMe technology can make server-based SDS higher performing and more
efficient
• IBM Storage has a roadmap that incorporates these technologies in both
SDS systems and in integrated appliances that combine software and
hardware
• Use cases provided as examples
34
Link:
http://www.redbooks.ibm.com/abstracts/redp5437.html?Open

Summary
35
NVMe is a relatively new block storage
protocol
• Meet the high performance demands
of flash storage
NVMe can be used inside servers and
storage devices, as well as over
networks
• NVMe-OF for Ethernet, IB
• FC-NVMe for Fibre Channel
Benefits include:
• Lower latency, More IOPS
• Fewer servers and software licenses
• Future-proof for Storage Class
Memory technologies

Please
complete
the session
survey!
36

37

For more Information on Flash
38
Product Pages:
– IBM FlashSystem 900
– IBM FlashSystem V9000
– IBM FlashSystem EcoSystem
Resources:
ibm.com/storage/flash/resources
Compression Estimator Tool: http://www-
304.ibm.com/webapp/set2/sas/f/comprestimator/h
ome.html
Workload Estimator Tool:
http://www.ibm.com/systems/support/tools/estimat
or
Energy Estimator Tool:
http://www.ibm.com/systems/support/tools/estimat
or/energy .
Twitter: #FlashSystem, #IBMFlash, #FlashRealized
Flash website: ibm.com/storage/flash/

IBM FlashWatch guarantees
39
The IBM FlashWatch programs demonstrate the confidence that IBM has in FlashSystem 9100:
1. Data reduction program designed to help reduce planning risks and help lower storage costs by
meeting baseline levels of data reduction effectiveness, up to 5:1 with FlashSystem 9100.
2. Controller upgrade program designed to help enable users of IBM FlashSystem 9100 to reduce
costs, while maintaining leading-edge controller technology for essentially the cost of ongoing
system maintenance.
3. 100% availability program designed to help enterprises avoid the costs and risks related to
business downtime by offering the availability of business-critical data and storage systems for your
company.
4. Flash endurance assurance program intended to help address concerns associated with using
even the most demanding workloads with IBM flash technology.
Plus …
An option to acquire 24×7 support for up to seven years.
Source: https://developer.ibm.com/storage/2018/07/10/peace-of-mind-with-ibm-flashwatch-program/

RDMA-enabled Ethernet Technologies for NVMe over Fabrics
iWARP - Internet Wide Area RDMA Protocol
• Implements RDMA over Internet Protocol networks
• Layered on a mix of layers including DDP (Direct Data Placement),
MPA (Marker PDU Aligned framing), RDMA protocol (RDMAP) over
TCP/IP
RoCE - RDMA over Converged Ethernet
• RoCE is based on InfiniBand transport over Ethernet
• RoCEv2 enhances RoCE with a UDP header and Internet routability
• Uses IP but not TCP
• Uses InfiniBand transport on top of Ethernet
• Requires a lossless DCB fabric
40
Connection
Technologies

Taxonomy of Transports - RoCE
• RoCE (pronounced rocky) is RDMA over Converged
Ethernet
• Two versions of RoCE
• Developed by the InfiniBand Trade Association
• Version 1 released in 2010 and is Internet protocol based and
limited to a broadcast domain similar limitation of FCoE
• Version 2 released in 2014 and is built on top of UDP (but in-order
delivery is not guaranteed) and packets between the same source
and destination pair must not be re-ordered by the network
• V2 has a simple congestion control mechanism
• Converged Ethernet with Data Center Bridging (DCB) networks are
needed to get the performance characteristics similar to InfiniBand
© Copyright IBM Corporation 2018. Technical University/Symposia materials may
not be reproduced in whole or in part without the prior written permission of IBM.
RDMA Software Stack
IB Transport Protocol
UDP/IP
Ethernet Link Layer
Ethernet / IP Management
RoCE v2
IBM Systems Technical Events | ibm.com/training/events
RoCE v1
RDMA Software Stack
IB Transport Protocol
IB Network Layer
Ethernet Link Layer
Ethernet Management

Taxonomy of Transports - iWARP
• iWARP stands for Internet Wide-Area RDMA Protocol
• Created by IETF and initially defined in 5 RFCs in 2007
(RFC 5040, RFC 5041, RFC 5042, RFC 5043 and RFC 5044)
• Since 2011, IETF has made some updates with 3 additional RFCs
(RFC 6580 - 2012, RFC 6581 - 2011 and RFC 7306 - 2014)
• Uses TCP (on top of UDP) or SCTP (stream control transmission
protocol) to control RDMA flows
• iWARP is a protocol, not a full implementation mechanism
• iWARP does not have any single standard programming interface but rather
defines protocol behavior in terms of the operations that are legal for the
protocol, known as Verbs
• Implemented as a part of TCP I/O stack (kernel/software bottleneck)
or via by using TOE NICs (TCP/IP Off-load Engine)
RDMA Software Stack
iWARP Protocol
TCP/IP Network
Ethernet Link Layer
Ethernet / IP Management
iWARP

Taxonomy of Transports - InfiniBand
• InfiniBand (IB) architecture specifications define how RDMA
over an IB network is performed
• IB has a link level flow control mechanism via a credit-based
algorithm that guarantees lossless communication
• IB has congestion control method based on FECN/BECN
(forward explicit congestion notification/backward explicit
congestion notification) marking
• IB switches have lower latency than Ethernet switches
(approximately 100 ns versus 230 ns), but cost per IB port is a
factor

FC-NVMe Introduction
• NVMe is mapped using the similar FCP mapping as SCSI
• Built from the ground up with non-volatile memory as the
storage to leverage the speed and robustness of Fibre
Channel
• Does not require building a parallel infrastructure
• Easily provides for scalability and access controls
• Aimed at 16Gbps, 32Gbps and higher speed switches and
fabrics
• The fabric switches have to be able to recognize FC-NVMe
services and devices plus handle the registration and
queries of FC features
• Brocade FOS 8.2 or higher
• Cisco NX_OS 8.x or higher

Taxonomy of Transports – FC-NVMe or FC-NVMe
• NVMe over Fibre Channel standards were developed by the T11
organization
• T11 standards documentation for NVMe is FC-NVMe
• Very similar to FCP (fibre channel protocol) where NVMe capsule is
“mapped” into the payload area of FC frame(s)
• FC-NVMe uses slightly more than 90% of in-host NVMe
implementation over a PCIe bus
• Main difference is NVMe-oF uses a message or packet model for
communication between a host and storage target, while in-host
NVMe implementations just use shared memory and memory
pointers
• NVMe-oF is meant to extend the distance between host and storage
device
• Original design goal of NVMe-oF was to add no more than 10
microseconds of latency

If you’re already FC, NVMe Adoption is easiest on FC
• Use existing infrastructure
• No surprises, no duplication of infrastructure or effort
• SCSI-based storage can be converted to NVMe-based storage
incrementally on App-by-App basis with easy roll back option
• Rely on known vendor relationship
• Shared vocabulary and trusted support models
• Build on robust Fibre Channel services
• Name services, discovery, zoning, flow control
• Leverage familiar tools and team expertize
• No need to start all over from scratch
• Get NVMe performance benefits while
migrating incrementally

FC-NVMe Components
• Same basic components as SCSI and FCP
Hosts (initiators), SAN (network) and storage devices (targets)
• Storage device can also be known as a NVMe Storage
Subsystem
• NVMe Storage System has:
• NVMe controllers (which contains the SQ and CQ queues)
• NVMe Namespace
• NVMe Storage Media
• NVMe controllers are logical devices which
can be created dynamically when a host
connects to a namespace or the NVMe Storage
Subsystem may have a static number of
controllers already set-up
• NVMe controllers handle tags to the namespace(s) which are
known as Namespace IDs (NSID)

FC-NVMe Components
FCP - SCSI-3 command set encapsulated in an FC frame
FC-NVMe - NVMe command set encapsulated in an FC frame
Substantial performance boost
because of:
• Command streamlining
• Reduced context switches
• Increased multithreading -
64,000 queues with a
maximum queue depth of
64,000

Summary of NVMe Connectivity Options
49
RDMA
Transport
NVMe Host Subsystem
NVMe Host-Side Transport Abstraction
Fibre Channel Transport NVMe RDMA
Fibre Channel
FC Fabric
Fibre Channel
NVMe Controller-SideTransport Abstraction
Fibre Channel Transport
NVMe Controller Subsystem
iWARP RoCE Infiniband
RDMA Fabric
RDMA Verbs
RDMA
Transport
NVMe RDMA
RDMA Verbs
iWARP RoCE Infiniband
NVMe PCIe
Host Software
PCIe Fabric
PCIe Function
NVME PCIe I/F Function

Additional Education
The State of Flash Storage and SCMs with IBM Fellow Andy Walls
• https://vts.inxpo.com/Launch/Event.htm?ShowKey=38406
If you want NVMe / NVMe-oF specification details
• http://www.nvmexpress.org/specifications/
• https://www.youtube.com/channel/UCc-fBI-VCJnRr-MoDTBffUQ
BrightTALK webinars
• https://www.brighttalk.com/search?q=nvme
Fibre Channel Industry Association FC-NVMe details
• http://fibrechannel.org/storage-fabric-options-with-nvme/
50

Session reference links
IBM Supports New Faster Protocols for Flash Storage
https://www-03.ibm.com/press/us/en/pressrelease/52280.wss
http://www.nvmexpress.org/wp-content/uploads/NVMe_Over_Fabrics.pdf
Todd Walls NVMe blog and video:
https://www.ibm.com/blogs/systems/looking-for-storage-performance-no-need-to-wait-for-nvme/
http://lt.be.ibm.com/stg/ltu64508
http://www-304.ibm.com/services/weblectures/div/partnerworld/ltu64508
http://www.ibm.com/training/events
51

NVMe-MI References
• NVMe/NVMe-MI - http://nvmexpress.org/
• RASM - https://software.intel.com/en-us/articles/rasm-a-primer-for-isv-applications-engineers
• RASM - http://www.ni.com/white-paper/14410/en/
• Manageability - http://www.ni.com/white-paper/14415/en/
• Reliability - http://www.ni.com/white-paper/14412/en/
• Serviceability - http://www.ni.com/white-paper/14414/en/
• Availability - http://www.ni.com/white-paper/14413/en/
52

IBM zHyperLink delivers better than NVMe-oF latencies for IBM Z today!
New storage technologies like flash storage are
driven by market requirements of low latency
which helps organizations to improve customer
satisfaction, generate revenue and address new
business opportunities
Low latency drove the high adoption rate of I/O
technologies including FICON Express16s,
SuperPAV and zHPF
zHyperLink delivers <20ms, TODAY!
IBM zHyperLink™ is the result of an IBM
research project created to provide extreme
low latency links between the mainframe and
the storage
IBM Z IBM
DS8880
Point to point interconnection
between the IBM Z Central
Electronics Complexes (CECs)
and the DS8880 I/O Bays
Less than 20µsec
response time !
53

About the Speaker
Tony Pearson is a Master Inventor and Senior IT Architect for the IBM Storage product line. Tony joined IBM Corporation in
1986 in Tucson, Arizona, USA, and has lived there ever since. In his current role, Tony presents briefings on storage topics
covering the entire IBM Storage product line, IBM Spectrum Storage software products, and topics related to Cloud Computing,
Analytics and Cognitive Solutions. He interacts with clients, speaks at conferences and events, and leads client workshops to
help clients with strategic planning for IBM’s integrated set of storage management software, hardware, and virtualization
solutions.
Tony writes the “Inside System Storage” blog, which is read by thousands of clients, IBM sales reps and IBM Business Partners
every week. This blog was rated one of the top 10 blogs for the IT storage industry by “Networking World” magazine, and #1
most read IBM blog on IBM’s developerWorks. The blog has been published in series of books, Inside System Storage: Volume
I through V.
Over the past years, Tony has worked in development, marketing and consulting for various storage hardware and software
products. Tony has a Bachelor of Science degree in Software Engineering, and a Master of Science degree in Electrical
Engineering, both from the University of Arizona. Tony is an inventor or co-inventor of 19 patents in the field of electronic data
storage.
9000 S. Rita Road
Bldg 9032 Floor 1
Tucson, AZ 85744
+1 520-799-4309 (Office)
tpearson@us.ibm.com
Tony Pearson
Master Inventor
Senior IT Architect
IBM Storage
54

Special Thanks for the following contributors to this presentation
Jim Blue,
Senior Software Engineer, IBM SAN Central / Switch PFE
Roger Kasten,
IBM Worldwide Flash Technical Enablement Manager
Brian Sherman
IBM Distinguished Engineer
55

56
IBM Tucson Client Experience Center
Tucson, Arizona is headquarters for
IBM storage hardware and software
design and development
IBM Tucson Client Experience
Center offers:
• Technology briefings
• Product demonstrations
• Solution workshops
• Lab tours
Take a video tour!
• http://youtu.be/CXrpoCZAazg
https://www.ibm.com/it-infrastructure/services/client-centers

57
Email:
tpearson@us.ibm.com
Twitter:
twitter.com/az990tony
Blog:
ibm.co/Pearson
Books:
www.lulu.com/spotlight/990_tony
IBM Expert Network on Slideshare:
www.slideshare.net/az990tony
Facebook:
www.facebook.com/tony.pearson.16121
LinkedIn:
https://www.linkedin.com/in/az990tony
Additional Resources from Tony Pearson

This presentation uses the IBM Plex™ font
58
IBM Plex™ is our new typeface. It’s global, it’s versatile and
it’s distinctly IBM.
IBM Plex
Sans
The IBM company is freeing itself from the cold, modernist cliché
and replacing Helvetica with a new corporate typeface. Also
replaces Arial, Calibri, Lucida Grande, Trebuchet, etc.
IBM Plex
Mono
A little something for developers. Replaces
Courier New, Letter Gothic, Lucida Console, etc.
IBM Plex
Serif
A hybrid of the third kind (combining the best of Plex, Bodoni,
and Janson into a contemporary serif). Replaces Cambria,
Garamond, Lucida Bright, Times New Roman, etc.
IBM Plex is freely available as TrueType and OpenType at: https://github.com/IBM/plex/releases

Notices and disclaimers
59
© 2018 International Business Machines Corporation. No part of this
document may be reproduced or transmitted in any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity. IBM products and
services are warranted per the terms and conditions of the agreements
under which they are provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are presented as
illustrations of how those
customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
References in this document to IBM products, programs, or services
does not imply that IBM intends to make such products, programs or
services available in all countries in which IBM operates or does
business.
Workshops, sessions and associated materials may have been
prepared by independent session speakers, and do not necessarily
reflect the views of IBM. All materials and discussions are provided
for informational purposes only, and are neither intended to, nor shall
constitute legal or other guidance or advice to any individual
participant or their specific situation.
It is the customer’s responsibility to insure its own compliance
with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any
relevant laws and regulatory requirements that may affect the
customer’s business and any actions the customer may need to take
to comply with such laws. IBM does not provide legal advice
or represent or warrant that its services or products will ensure that
the customer follows any law.
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products about
this publication and cannot confirm the accuracy of performance,
compatibility or any other claims related to non-IBM
products. Questions on the capabilities of non-IBM products should
be addressed to the suppliers of those products. IBM does not
warrant the quality of any third-party products, or the ability of
any such third-party products to interoperate with IBM’s products.
IBM expressly disclaims all warranties, expressed or implied,
including but not limited to, the implied warranties of
merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to,
and does not, grant any right or license under any IBM patents,
copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM
products and services used in the presentation] are trademarks of
International Business Machines Corporation, registered in many
jurisdictions worldwide. Other product and service names might
be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark
information" at: www.ibm.com/legal/copytrade.shtml.

S104878 nvme-revolution-jburg-v1809b

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie S104878 nvme-revolution-jburg-v1809b

Ähnlich wie S104878 nvme-revolution-jburg-v1809b (20)

Mehr von Tony Pearson

Mehr von Tony Pearson (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

S104878 nvme-revolution-jburg-v1809b