SG 245444
SG 245444
SG 245444
Octavian Lascu
Ewerson Palacio
Bill White
Redbooks
IBM Redbooks
September 2023
SG24-5444-22
Note: Before using this information and the product it supports, read the information in “Notices” on
page vii.
This edition applies to connectivity options that are available on the IBM z16 A01,IBM z16 A02, IBM z16 AGZ,
IBM z15 T01, IBM z15 T02, IBM z14 M0x, and IBM z14 Model ZR1.
© Copyright International Business Machines Corporation 1999, 2023. All rights reserved.
Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .x
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 I/O channel overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 I/O hardware infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 I/O connectivity features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 FICON Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 zHyperLink Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Open Systems Adapter-Express . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 HiperSockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6 Parallel Sysplex and coupling links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.7 Shared Memory Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8 I/O feature support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.9 Special-purpose feature support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.9.1 Crypto Express features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.9.2 Virtual Flash Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.9.3 IBM zEnterprise Data Compression Express feature . . . . . . . . . . . . . . . . . . . . . . 15
Contents v
9.3.1 Server Time Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
9.3.2 Dynamic split and merge for Coordinated Timing Network. . . . . . . . . . . . . . . . . 154
9.3.3 Operating system support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
9.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
AIX® IBM z14® VTAM®
CICS® IBM z16™ WebSphere®
Cognos® Parallel Sysplex® z/Architecture®
Db2® RACF® z/OS®
DS8000® Redbooks® z/VM®
FICON® Redbooks (logo) ® z/VSE®
GDPS® Resource Link® z13®
Guardium® S/390® z15®
IBM® System z® z16™
IBM Security® Tivoli® zEnterprise®
IBM Z® VIA®
The registered trademark Linux® is used pursuant to a sublicense from the Linux Foundation, the exclusive
licensee of Linus Torvalds, owner of the mark on a worldwide basis.
Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.
Red Hat, OpenShift, are trademarks or registered trademarks of Red Hat, Inc. or its subsidiaries in the United
States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Other company, product, or service names may be trademarks or service marks of others.
This IBM Redbooks® publication describes the connectivity options that are available for use
within and beyond the data center for the IBM Z® family of mainframes, which includes the
following systems:
IBM z16™ A01
IBM z16 A02
IBM z16 AGZ1
IBM z15® T01
IBM z15 T02
IBM z14® M01 - M05
IBM z14 ZR1
This book highlights the hardware and software components, functions, typical uses,
coexistence, and relative merits of these connectivity features. It helps readers understand
the connectivity alternatives that are available when planning and designing their data center
infrastructures.
The changes to this edition are based on the IBM Z hardware announcement that is dated
April 04, 2023.
This book is intended for data center planners, IT professionals, systems engineers, and
network planners who are involved in the planning of connectivity solutions for IBM®
mainframes.
Authors
This book was produced by a team of specialists from around the world working with IBM
Redbooks, Poughkeepsie Center.
Octavian Lascu is an IBM Redbooks Project Leader and a Senior IT Consultant with over
25 years of experience. He specializes in designing, implementing, and supporting complex
IT infrastructure environments (systems, storage, and networking), including high availability
and disaster recovery (HADR) solutions and high-performance computing (HPC)
deployments. He has developed materials for and taught over 50 workshops for technical
audiences around the world. He is the author of several IBM publications.
Ewerson Palacio is an IBM Redbooks Project Leader. He holds a bachelor’s degree in Math
and Computer Science. Ewerson worked for IBM Brazil for over 40 years and retired in 2017
as an IBM Distinguished Engineer. Ewerson co-authored many Redbooks publications about
IBM Z, and created and presented ITSO seminars around the globe.
Bill White is an IBM Redbooks Project Leader and Senior Infrastructure Specialist at IBM
Redbooks, Poughkeepsie Center.
1
The IBM z16 AGZ indicates a rack-mounted configuration that allows the core compute, I/O, and networking
features to be installed into and powered by a client-designated rack with power distribution units (PDUs). The
rack-mounted configuration options are under a combined AGZ warranty umbrella.
Markus Ertl, Jannie Houlbjerg, Hervey Kamga, Gerard Laumay, Slav Martinski,
Kazuhiro Nakajima, Martijn Raave, Paul Schouten, Anna Shugol, André Spahni, John Troy,
Roman Vogt, and Bo Xu
Robert Haimowitz
IBM Redbooks, Poughkeepsie Center
Bill Bitner, Patty Driever, Susan Farrell, Richard Gagnon, Darelle Gent, Les Geer III, Ron
Geiger, David Hutton, Tom Morris, Walter Niklaus, Purvi Patel, Franco Pinto, Eysha Powers,
Martin Recktenwald, Yamil Rivera, Lisa Schloemer, Christine Smith, Dean St Piere, Dave
Surman, Brian Valentine, Marna Walle, Barbara Weiler
IBM
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
Comments welcome
Your comments are important to us!
We want our books to be as helpful as possible. Send us your comments about this book or
other IBM Redbooks publications in one of the following ways:
Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
Send your comments in an email to:
redbooks@us.ibm.com
Mail your comments to:
IBM Corporation, IBM Redbooks
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400
Preface xi
xii IBM zSystems Connectivity Handbook
1
Chapter 1. Introduction
This chapter gives a brief overview of the input/output (I/O) channel architecture and
introduces the connectivity options that are available on IBM Z platforms.
Note: The link data rates that are described throughout this book do not represent the
actual performance of the links. The actual performance depends on many factors, which
include latency through the adapters and switches, cable lengths, and the type of workload
that uses the connection.
One of the many key strengths of the IBM Z platform is its ability to deal with large volumes of
simultaneous I/O operations. The channel subsystem (CSS) provides the function for IBM Z
platforms to communicate with external I/O and network devices and manage the flow of data
between those external devices and system memory. This goal is achieved by using a system
assist processor (SAP) that connects the CSS to the external devices.
The SAP uses the I/O configuration definitions that are loaded in the hardware system area
(HSA) of the system to identify the external devices and the protocol that they support. The
SAP also monitors the queue of I/O operations that are passed to the CSS by the operating
system.
By using an SAP, the processing units (PUs) are relieved of the task of communicating
directly with the devices, so data processing can proceed concurrently with I/O processing.
Increased system performance demands higher I/O and network bandwidth, speed, and
flexibility, so the CSS evolved with the advances in scalability of the IBM Z platforms.
z/Architecture provides functions for scalability in the form of multiple CSSs that can be
configured within the same IBM Z platform, for example:
IBM z16 A01, IBM z15 T01, and IBM z14 M0x support up to six CSSs.
IBM z16 A02, IBM z16 AGZ1, IBM z15 T02, and IBM z14 ZR1 support up to three CSSs.
All these IBM Z platforms deliver a significant increase in I/O throughput. For more
information, see Chapter 2, “Channel subsystem overview” on page 17.
1
The IBM z16 AGZ indicates a rack-mounted configuration that allows the core compute, I/O, and networking
features to be installed into and powered by a client-designated rack with power distribution units (PDUs). The
rack-mounted configuration options are under a combined AGZ warranty umbrella.
The Internal Coupling (IC) channel, IBM HiperSockets, and SMC can be used for
communications between logical partitions (LPARs) within the IBM Z platform.
The Open Systems Adapter (OSA) features provide direct, industry-standard Ethernet
connectivity and communications.
2
IBM z16, IBM z15, and IBM z14 ZR1 do not support InfiniBand coupling links. Careful planning is required if
IBM z14 M0x that uses InfiniBand coupling or timing links is part of a sysplex or Coordinated Timing Network
(CTN) configuration.
Chapter 1. Introduction 3
The 25GbE RoCE Express3 and 2.x and 10GbE RoCE Express3 and 2.x features provide a
high-speed, low-latency networking fabric for IBM Remote Direct Memory Access (RDMA)
communications (IBM z/OS® to z/OS, z/OS to Linux on IBM Z, and Linux on IBM Z to Linux
on IBM Z).
As part of system planning activities, you decide where to place the equipment (for distance
reasons), how it is operated and managed, and the business continuity requirements for
disaster recovery (DR), tape vaults, and so on. The types of software (operating systems and
applications) that are used must support the features and devices on the IBM Z platform.
From a hardware point of view, all the features in the PCIe+ I/O drawers are managed by the
IBM Z Support Elements (SEs). This function applies to installing and updating Licensed
Internal Code (LIC) to features and other operational tasks.
Many features have an integrated processor that handles the adaptation layer functions that
are required to present the necessary features to the rest of the system in a uniform manner.
Therefore, all the operating systems have the same interface with the I/O subsystem.
The IBM Z platform supports industry-standard PCIe adapters that are called native PCIe
adapters. For native PCIe adapter features, there is no adaptation layer, but the device driver
is present in the operating system. The adapter management functions (such as diagnostics
and firmware updates) are provided by Resource Groups.
Four Resource Groups are available on IBM z16, IBM z15, and IBM z14. The Resource
Groups are managed by an integrated firmware processor (IFP) that is part of the system’s
base configuration.
The following sections briefly describe connectivity options for the I/O features that are
available on the IBM Z platforms.
IBM Z platforms build on this I/O architecture by offering high-speed FICON connectivity, as
shown in Figure 1-1 on page 5.
The FICON implementation enables full-duplex data transfer. In addition, multiple concurrent
I/O operations can occur on a single FICON channel. FICON link distances can be extended
by using various solutions. For more information, see 10.2.2, “FICON repeated distance
solutions” on page 161.
The FICON features on IBM Z also support full fabric connectivity for the attachment of SCSI
devices by using the FCP. Software support is provided by IBM z/VM®, IBM z/VSE® (SCSI
disk devices), Linux on IBM Z, and the KVM hypervisor.
The zHyperLink Express feature is a PCIe adapter that can be shared by multiple LPARs.
Chapter 1. Introduction 5
The zHyperLink Express feature is installed in the PCIe I/O drawer. On the IBM DS8000 side,
the fiber optic cable connects to a zHyperLink PCIe interface in an I/O bay.
The zHyperLink Express feature has the same qualities of service, as do all IBM Z I/O
channel features.
The OSA-Express features bring the strengths of the IBM Z family, such as security,
availability, and enterprise-wide access to data to the LAN environment. OSA-Express
provides connectivity for the following LAN types:
1000BASE-T Ethernet (100/1000 Mbps4)
1 Gbps Ethernet SR and LR
4 OSA Express7s 1000BASE-T and OSA Express7S 1.2 1000BASE-T support only 1000 Mbps full duplex.
1.5 HiperSockets
IBM HiperSockets technology provides seamless network connectivity to consolidate virtual
servers in an advanced infrastructure intraserver network. HiperSockets creates multiple
independent and integrated virtual local area networks (VLANs) within an IBM Z platform.
The z/VM virtual switch can transparently bridge a guest virtual machine network connection
on a HiperSockets LAN segment. This bridge allows a single HiperSockets guest virtual
machine network connection to directly communicate with other guest virtual machines on the
virtual switch and with external network hosts through the virtual switch OSA UPLINK port.
Chapter 1. Introduction 7
Figure 1-4 shows an example of HiperSockets connectivity with multiple LPARs and
virtual servers.
z/VM LP-1
Sysplex A
Linux Linux Linux z/VM
Guest 1 Guest 2 Guest n TCP/IP
z/OS z/OS z/OS
LP-5 LP-6 LP-7
Sysplex traffic
isolated from
non-sysplex
Application HiperSockets FE traffic
traffic FD
separation
Different sysplex
FF traffic on same
server
Linux Linux Linux
LP-2 LP-3 LP-4
FC Sysplex B
Figure 1-4 HiperSockets connectivity: Multiple LPARs and virtual servers (FC, FD, FE, and FF are
CHPIDs)
5
IBM z14 M0x is the last server that supports InfiniBand coupling links. Coupling links to other IBM z15 and IBM z16
can use only CS5 and CL5 CHPID types.
IBM Z
z/OS
CF01
ICF
IBM Z IBM Z
z/OS
z/OS
IC
CF02
ICF
FICON
The systems in a Parallel Sysplex configuration are linked and can fully share devices and run
the same applications. This feature enables you to harness the power of multiple IBM Z
platforms as though they are a single logical computing system.
The architecture is centered on the implementation of a CF that runs the Coupling Facility
Control Code (CFCC) and high-speed coupling connections for intersystem and intrasystem
communications. The CF provides high-speed data sharing with data integrity across multiple
IBM Z platforms.
Parallel Sysplex technology provides high availability (HA) for business-critical applications.
The design is further enhanced with the introduction of System-Managed Coupling Facility
Structure Duplexing, which provides the following extra benefits:
Availability: Structures do not need to be rebuilt if a CF fails.
Manageability and usability: A consistent procedure is established to manage structure
recovery across users.
Configuration benefits: A sysplex can be configured with internal CFs only.
6
There are many configurations that are possible for Parallel Sysplex, which depend on the available hardware and
clustering requirements.
Chapter 1. Introduction 9
Attention: Parallel Sysplex technology and an STP Coordinated Timing Network (CTN)
network support connectivity between systems that differ by up to two generations (n-2).
For example, an IBM z16 can participate in Parallel Sysplex cluster with IBM z15 and
IBM z14.
The IBM z16 does not support InfiniBand. You can set up connectivity by using only
PCIe-based coupling, such as the Integrated Coupling Adapter Short Reach (ICA SR) and
CE LR features.
SMC allows two peers to send and receive data by using system memory buffers that each
peer allocates for its partner’s use. Two types of SMC protocols are available on the IBM Z
platform:
SMC-Remote Direct Memory Access (SMC-R) Version 1 (SMC-Rv1) and Version 2
(SMC-Rv2).
SMC-R is a protocol for RDMA communication between TCP socket endpoints in LPARs
in different systems. SMC-R runs over networks that support RDMA over Converged
Ethernet (RoCE). It allows existing TCP applications to benefit from RDMA without
requiring modifications. SMC-R provides dynamic discovery of the RDMA capabilities of
TCP peers and automatic setup of RDMA connections that those peers can use.
The 25GbE RoCE Express3, 25GbE RoCE Express2.1, 25GbE RoCE Express2,
10GbE RoCE Express3, 10GbE RoCE Express2.1 and 10GbE RoCE Express2 features
provide the RoCE support that is needed for LPAR-to-LPAR communication across IBM Z
platforms.
While SMC-Rv1 does not support routing (communication across multiple subnets),
SMC-Rv2 updates to the SMC-R protocol allowing SMC-Rv2 enabled hosts to connect
and communicate across multiple IP subnets.
SMC-Direct Memory Access (SMC-D)
SMC-D implements the same SMC protocol that is used with SMC-R to provide highly
optimized intra-system communications. Where SMC-R uses RoCE for communicating
between TCP socket endpoints in separate systems, SMC-D uses Internal Shared
Memory (ISM) technology for communicating between TCP socket endpoints in the same
IBM Z platform.
ISM provides adapter virtualization (virtual functions (VFs)) to facilitate the intra-system
communications. Hence, SMC-D does not require any additional physical hardware (no
adapters, switches, fabric management, or PCIe infrastructure).
Therefore, significant cost savings can be achieved when using the ISM for LPAR-to-LPAR
communication within the same IBM Z platform.
zHyperLink 0431 32 16 32 16 16 16 2
Chapter 1. Introduction 11
Feature Maximum number of ports Ports
I/O feature Code per
IBM z16 IBM z16 IBM z15 IBM z15 IBM z14 IBM z14 feature
A01 A02/AGZ T01 T02 M0x ZR1
OSA-Express7S 25GbE
0449 48 N/A 48 N/A N/A N/A 1
SR1.1
Coupling links Chapter 9, “Coupling links and common time” on page 141
IC N/A 64 64 64 64 32 32 N/A
ICA SR 0172 96 48 96 48 80 16 2
Chapter 1. Introduction 13
1.9 Special-purpose feature support
In addition to the I/O connectivity features, several special-purpose features are available that
can be installed in the PCIe+ I/O drawers, such as the following ones:
Crypto Express
IBM zEnterprise® Data Compression (zEDC) (IBM z14 only)
The new Crypto Express8S is the first adapter that supports the new Quantum Safe function,
and it also is the first one that is designed for Federal Information Processing Standards
(FIPS) 140-3 Level 4.
Quantum-safe cryptography refers to efforts to identify algorithms that are resistant to attacks
by both classical and quantum computers to keep information assets secure even after a
large-scale quantum computer is built.
Crypto Express8S, Crypto Express7S, and Crypto Express6S are tamper-sensing and
tamper-responding programmable cryptographic features that provide a secure cryptographic
environment. Each adapter contains a tamper-resistant Hardware Security Module (HSM).
The HSM can be configured as a secure IBM Common Cryptographic Architecture (CCA)
coprocessor, as a secure IBM Enterprise PKCS #11 (EP11) coprocessor, or as an
accelerator.
In IBM z15 and IBM z16, the compression capability is integrated into the processor chip. The
Integrated Accelerator for zEDC provides hardware-based acceleration of data compression
and decompression, which improves cross-platform data exchange, reduces processor use,
and saves disk space. The Integrated Accelerator for zEDC implements compression as
defined by RFC1951 (DEFLATE). For more information about the DEFLATE Compress Data
Format Specification, see RFC for DEFLATE Compressed Data Format Specification Version
1.3.
On IBM z14, a zEDC Express PCIe feature can be shared by up to 15 LPARs. z/OS 2.2 and
later supports the zEDC Express feature. z/VM 7.1 with program temporary fixes (PTFs) and
later provides guest exploitation. With IBM z15 and later systems, the Integrated Accelerator
for zEDC is integrated onto the processor chip, so there is no virtualization requirement.
Table 1-2 lists the special-purpose features that are available on IBM Z platforms. Not all
special-purpose features can be ordered on all systems, and certain features are available
only with a system upgrade. All special-purpose features are installed in the PCIe I/O drawer
or the PCIe+ I/O drawer.
zEDC Express
Chapter 1. Introduction 15
16 IBM zSystems Connectivity Handbook
2
Channel path
A channel path is a single interface between a system and one or more CUs. Commands and
data are sent across a channel path to process I/O requests. A CSS can have up to 256
channel paths.
Subchannels
A subchannel provides the logical representation of a device to the program and contains the
information that is required to sustain a single I/O operation. One subchannel is assigned for
each device that is defined to the LPAR. Subchannel set (SS) availability per platform is
shown in the following list:
Four SSs are available on IBM z16 A01, IBM z15 T01, and IBM z14 M0x.
Three SSs are available on IBM z16 A02, IBM z16 AGZ, IBM z15 T02, and IBM z14 ZR1.
With IBM Z platforms, a CHPID number is assigned to a physical location (slot or port) by the
user through either the Hardware Configuration Definition (HCD) or the Input/Output
Configuration Program (IOCP).
Control units
A CU provides the logical capabilities that are necessary to operate and control an I/O device.
It adapts the characteristics of each device so that it can respond to the standard form of
control that is provided by the CSS. A CU can be housed separately, or it can be physically
and logically integrated with the I/O device, the CSS, or in the system itself.
I/O devices
An I/O device provides external storage, a means of communication between data processing
systems, or a means of communication between a system and its environment. In the
simplest case, an I/O device is attached to one CU and accessible through one channel path.
Table 2-1 lists the maximum number of CSSs and LPARs that are supported by IBM Z
platforms. CSSs are numbered 0 - 5 or 0 - 2 (system dependent), with the numbers referred
to as CSS image IDs.
Table 2-1 Maximum number of CSSs and LPARs that are supported by IBM Z
Systems Maximum number of CSSs Maximum number of LPARs
IBM z14 6 85
For efficient management, consider using the HCD tool to build and control the IBM Z
input/output configuration definitions. HCD support for multiple CSSs is available with
IBM z/VM and IBM z/OS. HCD provides the capability to make dynamic I/O configuration
changes.
LPARs cannot be added until at least one CSS is defined. LPARs are defined for a CSS, not
for a system. An LPAR is associated with only one CSS. CHPID numbers are unique within a
CSS, but the same CHPID number can be reused within all CSSs.
All CSSs are defined within a single I/O configuration data set (IOCDS). The IOCDS is loaded
into the hardware system area (HSA) and initialized during power-on reset.
On the IBM Z platform, the HSA has a fixed size, which is not included in the purchased
memory. Table 2-2 lists the HSA sizes
CHPIDs are not preassigned on IBM Z platform. Assign the CHPID numbers by using the
CHPID Mapping Tool or directly by using HCD or IOCP. Assigning CHPIDs means that the
CHPID number is associated with a physical channel port location (PCHID) or the adapter ID
(AID) and a CSS. The CHPID number range is 00 - FF, and must be unique in a CSS. Any
CHPID that is not connected to a PCHID fails validation when an attempt is made to build a
production input/output definition file (IODF) or an IOCDS.
Example 2-1 shows a portion of a sample PCHID REPORT for an IBM z16 A01 system.
For more information about recommendations for configuring CUs and devices to best
leverage the reliability, availability, and serviceability (RAS) and performance of the processor,
see IOCP Users Guide, which is available from IBM Resource Link1. The CHPID Mapping
tool, which also is available from IBM Resource Link®, might help you configure channels to
CUs and devices.
1
A user ID is required to access the IBM Resource Link website. To obtain a user ID, follow the instructions that are
provided at the website.
You can get a PCHID report from the IBM account team.
2.1.5 Adapter ID
The AID number assigns a CHPID to a port by using HCD or IOCP for IBM Parallel Sysplex
cluster technology.
On the IBM Z platform, the AID is bound to the serial number of the fanout. If the fanout is
moved, the AID moves with it. No IOCDS update is required if adapters are moved to a new
physical location.
AIDs are included in the PCHID report that IBM provides for new build (NB) systems and
upgrades. Example 2-2 shows an AID in a PCHID report.
LG13-LG16 0C-0F
(InfiniBand)
LG13-LG16 08-0B
(InfiniBand)
LG13-LG16 04-07
(InfiniBand)
LG13-LG16 00-03
(InfiniBand)
a. Indicates the IBM z14 physical central processor complex (CPC) drawer installation order.
b. The designation between the parenthesis indicates the logical CPC drawer number.
Table 2-4 lists the AID numbers for an IBM z14 ZR1.
LG07 - LG10 14 - 17
Table 2-5 lists the AID numbers for an IBM z15 T01.
LG01 00 0C 18 24 30
LG02 01 0D 19 25 31
LG03 02 0E 1A 26 32
LG04 03 0F 1B 27 33
LG05 04 10 1C 28 34
LG06 05 11 1D 29 35
LG07 06 12 1E 2A 36
LG08 07 13 1F 2B 37
LG09 08 14 20 2C 38
LG10 09 15 21 2D 39
LG11 0A 16 22 2E 3A
LG12 0B 17 23 2F 3B
Table 2-6 lists the AID numbers for an IBM z15 T02.
Table 2-7 lists the AID numbers for an IBM z16 A01.
LG01 00 0C 18 24
LG02 01 0D 19 25
LG03 02 0E 1A 26
LG04 03 0F 1B 27
LG05 04 10 1C 28
LG06 05 11 1D 29
LG07 06 12 1E 2A
LG08 07 13 1F 2B
LG09 08 14 20 2C
LG10 09 15 21 2D
LG11 0A 16 22 2E
LG12 0B 17 23 2F
Table 2-8 lists the AID numbers for an IBM z16 A02.
Figure 2-1 shows two CSSs that are defined as CSS0 and CSS1.
CHPIDs 80 81 90 91 80 81 90 91
Directors
The CHPIDs are mapped to their designated PCHIDs by using the IBM CHPID Mapping Tool
or manually by using HCD or IOCP. The output of the CHPID Mapping Tool is used as input to
HCD or the IOCP to establish the CHPID-to-PCHID assignments.
Spanning is the ability for the channel to be configured to multiple CSSs. When so defined,
the channels can be transparently shared by any or all of the configured LPARs, regardless of
the CSS to which the LPAR is configured.
A channel is considered a spanned channel if the same CHPID number in different CSSs is
assigned to the same PCHID in the IOCP or is defined as spanned in HCD. The same
situation applies to internal channels such as IBM HiperSockets technology, but there is no
PCHID association. Internal channels are defined with the same CHPID number in multiple
CSSs.
If all CHPIDs are spanned across multiple CSSs, only 256 channels can be supported.
Table 2-9 shows which channel types can be defined as shared or spanned channels.
Subchannel numbers, including their implied path information to devices, are limited to four
hexadecimal digits by hardware and software architectures. Four hexadecimal digits provide
64,000 addresses, which are known as a set. IBM reserves 256 subchannels, leaving 63,750
subchannels for general use with the IBM Z platform. Parallel access volumes (PAVs) make
this limitation of subchannels a challenge for larger installations. A single-disk drive (with
PAVs) often uses at least four subchannels.2
It was difficult to remove this constraint because the usage of four hexadecimal digits for
subchannels and device numbers that correspond to subchannels is specified in several
places. Expanding the field would break too many programs.
The solution allows sets of subchannels (addresses) with a current implementation of four sets
with IBM z16 A01, IBM z15 T01, IBM z14 M0x, and three sets with IBM z16 A02, IBM z16
AGZ, IBM z15 T02, and IBM z14 ZR1. Each set provides 64K (64,536) addresses minus one.
SS 0, the first set, reserves 256 subchannels for use by IBM (65,280 devices (64K-256 or
63.75K)). SSs 1 - 3 provide a full range of 64K minus one subchannel on the IBM Z platform
(65,535 devices (64K-1)).
The first SS (SS0) allows definitions of any type of device that is supported, for example,
bases, aliases, secondaries, and devices, other than disks that do not implement the concept
of associated aliases or secondaries.
The second, third, and fourth SSs (SS1, SS2, and SS3) are designated for use for disk alias
devices, both primary and secondary devices, and for IBM Metro Mirror secondary devices
only.
There is no required correspondence between addresses in the three sets. For example, it is
possible to have device number 8000 in SS 0 and device number 8000 in SSs 1 or 2, and they
can refer to different devices. Likewise, device number 1234, SS 0, and device number 4321,
SS 1, might be the base and an alias for the same device.
2 Four SSs are mostly used with PAV. They represent base addresses and three alias addresses.
The appropriate SS number must be included in IOCP definitions or in the HCD definitions
that produce the IOCDS. The SS number defaults to zero, and IOCP changes are needed
only when using SSs 1, 2, or 3.
This feature allows the use of Metro Mirror (Peer-to-Peer Remote Copy (PPRC)) secondary
devices. These devices are defined by using the same device number and a new device type
in an alternative SS to be used for IPL, IODF, and stand-alone memory dump volumes when
needed.
Running an IPL from an alternative SS is supported on the IBM Z platforms. It applies to the
IBM FICON and IBM High-Performance FICON for IBM Z (zHPF) protocols.
2.1.9 Summary
Table 2-10 lists the maximum number of CSS elements that are supported per IBM Z
platform.
Partitions 15 for the first two CSSs and 10 for 15 for the first 5 CSSs and 10 for the sixth,
the third, up to 40 per system up to 85 per system
CHPIDs 256 per CSS, up to 768 per system 256 per CSS, up to 1536 per system
Note: The information in this section applies to systems that are configured to run in
IBM Processor Resource/Systems Manager (PR/SM) mode. For systems running in IBM Z
Dynamic Partition Manager (DPM) mode, I/O configuration is managed trough the DPM
GUI that is available on the Hardware Management Console (HMC).
CSS controls communication between a configured channel, the CU, and the device. The
IOCDS defines the channels, CUs, and devices to the designated LPARs within the system.
This communication is defined by using the IOCP.
The IOCP statements typically are built by using the HCD. An interactive dialog is used to
generate the IODF, start the IOCP program, and then build the production IOCDS. The
IOCDS is loaded into the HSA and initialized during power-on reset. In earlier IBM Z servers,
the HSA storage was allocated based on the size of the IOCDS, partitions, channels, CUs,
and devices. Extra storage was reserved for dynamic I/O reconfiguration, if enabled.
With the IBM Z platform, CHPIDs are mapped to PCHIDs or AIDs by using the configuration
build process through HCD or IOCP.
The sections that follow describe tools that are used to maintain and optimize the I/O
configuration on IBM Z platform.
Consider using the mapping tool for all new builds of IBM Z family or when upgrading from
one system to another system. You also can use it as part of standard hardware changes (for
example, miscellaneous equipment specification (MES)3) for an existing IBM Z.
The mapping tool does not automatically map CS5 or CIB CHPIDs to AIDs and ports. This
process must be done manually, either in HCD, IOCP, or the mapping tool. The mapping tool
provides availability intersects for defined CIB CHPIDs. For more information about the
CHPID Mapping Tool, see IBM Documentation.
For a system configuration, the IBM Z configurator build process includes all physical
resources that are required for a particular I/O configuration, based on the supplied channel
type and quantity requirements.
The definition phase starts after the physical resources are ordered. The channels must be
defined according to the architecture’s rules, the system’s implementation, and the order.
The operational characteristics of a particular channel type, along with the addressing
capabilities, can affect configuration complexity, topology, infrastructure, and performance.
All CUs and I/O devices that attach to the system must be defined to a CSS. Specify the
following items when defining the I/O configuration for the system through HCD/IOCP:
LPARs (LPAR name, CSS ID, and MIF ID, where appropriate)
Channel paths on the system, their assignment to CSSs, and LPARs
FICON Directors (where appropriate)
CUs that are attached to the channel paths
I/O devices that are assigned to the CUs
Per FC 32K
For more information about CHPID types and channel configuration rules, see Input/Output
Configuration Program User’s Guide for ICP IOCP, SB10-7177.
For more information about how to configure a Stand-alone CF CPC, Linux on Z and zTPF
CPCs using HCD, refer to 9.2.6, “Dynamic I/O reconfiguration for stand-alone CF, Linux on Z
and z/TPF CPCs” on page 148.
Note: Not all FICON Express features are supported on all IBM Z platforms. Support for
each feature is listed in Table 3-2 on page 59.
The FICON implementation enables full-duplex data transfer, so data travels in both directions
simultaneously. FICON also enables multiple concurrent I/O operations.
Terminology: Throughout this chapter, FICON refers to the FICON Express8S, FICON
Express16S, FICON Express16S+, FICON Express16SA, and FICON Express32S
features, except when the function that is described is applicable to a specific feature.
The FICON channel matches data storage and access requirements with the latest
technology in servers, control units (CUs), and storage devices. FICON channels allow faster
and more efficient data transfer while allowing you to use their currently installed single-mode
(SM) and multimode (MM) fiber optic cabling plant.
FICON uses long wavelength (LX) and short wavelength (SX) transceivers with SM and MM
fiber optic media for data transmission.
With IBM Z, point-to-point connections can be used to access data that is stored on devices
without using an FC switch. In addition, an operating system or other stand-alone program
can undergo an IPL through a point-to-point connection by using the SCSI IPL feature.
N_Port ID Virtualization (NPIV) is not supported by FCP point to point. For more information,
see “Worldwide port name tool” on page 39.
The FCP support allows z/VM, Linux on IBM Z and the KVM hypervisor, and z/VSE operating
systems to access industry-standard SCSI devices. For disk applications, these FCP storage
devices use fixed block sectors rather than the Extended Count Key Data (ECKD) format.
FCP channel full fabric support means that multiple numbers of directors or switches can be
placed between the IBM Z platform and the SCSI device. This technique enables many hops
through a storage area network (SAN) and provides improved use of intersite-connected
resources and infrastructure. This expanded ability to attach storage devices provides more
choices for storage solutions and the ability to use existing storage devices. This configuration
can facilitate the consolidation of UNIX server farms onto the IBM Z platform, which protects
investments in SCSI-based storage.
For a list of switches, storage controllers, and devices that are verified to work in an FC
network that is attached to FCP channel, and the software requirements to support FCP and
SCSI controllers or devices, see the I/O Connectivity website.
FICON channels in FCP mode are based on the FC standards that are defined by INCITS
and published as ANSI standards. FCP is an upper-layer FC mapping of SCSI on a common
stack of FC physical and logical communication layers.
SCSI is supported by a wide range of controllers and devices, complementing the classical
storage attachment capability through FICON channels. FCP is the base for
industry-standard FC networks or SANs.
FC networks consist of servers, storage controllers, and devices as end nodes, which are
interconnected by FC switches, directors, and hubs. Switches and directors are used to build
FC networks or fabrics. Fibre Channel Arbitrated Loops (FC-ALs) can be constructed by
using FC hubs. In addition, different types of bridges and routers can be used to connect
devices with different interfaces, such as parallel SCSI. All these interconnections can be
combined in the same network.
SCSI is implemented by many vendors in many different types of storage controllers and
devices. These controllers and devices are widely accepted in the marketplace and have
proven to be able to meet the reliability, availability, and serviceability (RAS) requirements of
many environments.
FICON channels in FCP mode use the queued direct input/output (QDIO) architecture for
communication with the operating system. The QDIO architecture for FCP channels derives
from the QDIO architecture, which was defined initially for the OSA-Express features and
HiperSockets communications.
FCP channels do not use control devices. Instead, data devices that represent QDIO queue
pairs (QPs) are defined, and they consist of a request queue and a response queue. Each QP
represents a communication path between an operating system and the FCP channel. A QP
allows an operating system to send FCP requests to the FCP channel through the request
queue. The FCP channel uses the response queue to pass completion indications and
unsolicited status indications to the operating system.
Figure 3-3 shows the necessary FCP I/O definitions and compares them to FICON I/O
definitions.
Control Controller
Unit Device Device
Each FCP channel can support up to 480 QDIO QPs with the IBM Z platform. This support
allows each FCP channel to be shared among 480 operating system instances (with a
maximum of 252 guests per LPAR).
Host operating systems that share access to an FCP channel can establish up to 2048
concurrent connections to up to 512 different remote FC ports that are associated with FC
controllers. The total number of concurrent connections to end devices, which are identified
by LUNs, must not exceed 4096.
Although multiple operating systems can concurrently access the same remote FC port
through a single FCP channel, FC devices, which are identified by their LUNs, can be reused
only serially. For two or more unique operating system instances to share concurrent access
to a single FC or SCSI device (LUN), each of these operating systems must access this
device through a different FCP channel.
WWPN tool: The WWPN tool is supported in IBM Processor Resource/Systems Manager
(PR/SM) mode. DPM uses a different assignment algorithm for generating and assigning
NPIV WWPNs.
The WWPN tool can calculate and show WWPNs for both virtual and physical ports ahead of
system installation. This feature means that the SAN configuration can be retained rather
than altered by assigning a WWPN to physical FCP ports when a FICON feature is replaced.
The WWPN tool takes an adhesive file that contains the FCP-specific I/O device definitions
and creates the WWPN assignments that are required to set up the SAN. A binary
configuration file that can be imported later by the system is also created. The CSV (.csv) file
can be either created manually or exported from the HCD or Hardware Configuration
Manager (HCM).
The WWPN tool can be downloaded from the Tools section of IBM Resource Link (requires
registration).
The IPL device is identified by its SAN address, which consists of the WWPN of the disk
controller and the LUN of the IPL device.
Important: If a second-level z/VM system undergoes an IPL from an FCP SCSI LUN, a
minimum virtual memory is required, which depends on the model of the processor on
which the z/VM system is running. To ensure success on all processor models, you should
define at least 768 MB of virtual storage.
A stand-alone-dump program can also undergo an IPL from an FCP channel that is attached
to a SCSI disk. The stand-alone-dump program can also store the generated dumped data on
a SCSI disk. z/VM support of SCSI IPL allows Linux and other guest operating systems that
support this feature to undergo an IPL from an FCP-attached SCSI disk when z/VM is running
on an IBM Z platform. Therefore, Linux guests can be started and run from an FCP
channel-attached disk.
Multipathing over FCP is different. With FCP multipathing on Linux on IBM Z, each path to
each LUN appears to the operating system as a separate device. For example, if there are
four paths to five LUNs, the Linux kernel defines 20 SCSI devices.
At the time of writing, supported distributions use device-mapper multipathing in the Linux
kernel along with multipath-tools in the user space. For more information, see the
corresponding distribution documentation and How to use FC-attached SCSI devices.
I/O devices
The IBM Z FCP channel implements the FCP standard as defined by the INCITS Fibre
Channel Protocol for SCSI (FC-FCP) and Fibre Channel Protocol for SCSI Second Version
(FCP-2), and the relevant protocols for the SCSI-2 and SCSI-3 protocol suites. Theoretically,
each device that conforms to these protocols works when attached to an IBM Z FCP channel.
However, experience shows that there are small deviations in the implementations of these
protocols.
Also, for certain types of FCP and SCSI controllers and devices, specific drivers in the
operating system might be required to use all capabilities of the controller or device. The
drivers might also be required to cope with unique characteristics or deficiencies of the
device.
These hardware assists allow a cooperating guest operating system to start QDIO operations
directly to the applicable channel without interception by z/VM, which provides more
performance improvements. This support is integrated into the IBM Z platform. Consult the
appropriate Preventive Service Planning (PSP) buckets (3931DEVICE, 8561DEVICE,
3906DEVICE, 3932DEVICE, 8562DEVICE, or 3907DEVICE) before implementation.
An extension to the standard that is called Data Integrity Extensions (DIX) provides checksum
protection from the application layer through the host bus adapter (HBA), where cyclical
redundancy check (CRC) protection is implemented.
T10-DIF support by the FICON features, when defined as CHPID type FCP, is available on
the IBM Z platform. Using the T10-DIF standard requires support by the operating system
and the storage device.
Note: NPIV is supported in a switched topology, and FCP with NPIV is not supported in
a point-to-point topology.
Switch topology
FCP channels support full fabric connectivity, which means that several directors or
switches can be used between a IBM Z platform and the device. With the FICON
cascaded director support, the FICON storage network topology is limited to a
two-director, single-hop configuration.
Enterprise fabric
The usage of cascaded FICON Directors ensures the implementation of a high-integrity
fabric. For FCP, a high-integrity fabric solution is not mandatory, although it must be
considered. For example, if an FCP Inter-Switch Link (ISL) must be moved, data might
potentially be sent to the wrong path without notification. This type of error does not
happen on an enterprise fabric with FICON.
Transmission data checking
When a transmission is sent through an FCP channel, because of its full fabric capability,
FCP checks data for each leg of that transmission. FICON also checks intermediate data.
FICON Express16SA, FICON Express16S+, and FICON Express16S support FEC coding on
top of their 64 b/66 b data encoding for 16 Gbps connections. Their FEC design can correct
up to 11 bit errors per 2112 bits that are transmitted. FICON Express32G uses 256b/257b
encoding and can correct up to 20 bit errors per 5140 bits that are transmitted. Thus, when
connected to devices that support FEC at 16 or 32 Gbps connections, the FEC design allows
FICON Express channels to operate at higher speeds over longer distances and with reduced
power and higher throughput. Concurrently, the FEC design maintains the same reliability and
robustness that FICON channels are known for.
With IBM DS8870 or later, the IBM z16, IBM z15, IBM z14, and IBM z14 ZR1 can extend the
usage of FEC to the fabric N_Ports1 for a completed end-to-end coverage of 16 or 32 Gbps
FC links. For more information, see IBM DS8900F and IBM Z Synergy DS8900F: Release 9.3
and z/OS 2.5, REDP-5186.
1 Node ports
FIDR can be enabled by defining dynamic routing capable switches and CUs in HCD. Also,
z/OS has implemented a health check function for FIDR.
FICON performance
For more information about FICON and FCP performance, see the IBM server connectivity
web page.
Table 3-1 Fiber optic cabling for FICON: Maximum distances and link loss budget
FC-PI-4 Cable 2 Gbps 4 Gbps 8 Gbps 16 Gbps 32 Gbps 10 Gbps
Fiber core type ISLa
50 µm MM OM4 500 m / 3.31 400 m / 2.95 190 m / 2.19 125 m / 1.95 100 m / 1.86 N/A
50 µm MM OM3 500 m / 3.31 380 m / 2.88 150 m / 2.04 100 m / 1.86 70 m / 1.75 300 m / 2.6
Note: IBM does not support a mix of 50 µm and 62.5 µm fiber optic cabling in the same
physical link.
Then, the IOS issues a start subchannel (SSCH) instruction with the subsystem identifier
(SSID) that represents the device and the ORB as operands. The CSS is signaled to perform
the operation. This flow is shown in Figure 3-4 on page 47.
FICON CU
The most appropriate FICON channel is selected by the CSS. The FICON channel fetches
channel programs (CCWs) that are prepared by the application, fetches data from memory
(write) or stores data into memory (read), and presents the status of the operation to the
application (I/O interrupt).
The z/Architecture channel commands, data, and status are packaged by the FICON channel
into FC-SB-3 or FC-SB-4 (FC-4 layer) Information Units (IUs). IUs from several different I/O
operations to the same or different CUs and devices are multiplexed or demultiplexed by the
FC-2 layer (framing). These FC-2 frames with encapsulated FC-SB-3 or FC-SB-4 IUs are
encoded or decoded by the FC-1 layer (encode or decode) and sent to or received from the
FC-0 fiber optic medium (optics).
CCW4 CCW4
CCW5 CCW5
CCW6 CCW6
----- -----
CE/DE
Status
Except
Figure 3-5 CCW chaining
Using zHPF with the FICON channel, the z/OS operating system, and the CU reduces the
FICON channel overhead. This goal is achieved by protocol optimization and reducing the
number of IUs processed, which results in more efficient usage of the fiber link.
The mode that is used for an I/O operation depends on the CU that is supporting zHPF and
the settings in the z/OS operating system. An IECIOSxx parameter and SETIOS commands in
z/OS can enable or disable zHPF. Support is also added for the D IOS, ZHPF system
command to indicate whether zHPF is enabled, disabled, or not supported on the system.
During link initialization, both the channel and the CU indicate whether they support zHPF.
The Process Login (PRLI) support indicator is presented in response to the RNID Extended
Link Services (ELS). If PRLI is supported, the channel sends a PRLI ELS. Then, the PRLI
response indicates that zHPF is supported by the CU.
The way that zHPF transport mode manages CCW operation is different from the CCW
operation for the existing FICON architecture command mode, as shown in Figure 3-6. In
command mode, each single CCW is sent to the CU for execution. In transport mode, all
CCWs are sent over the link in one single frame to the CU. Certain complex CCW chains are
not supported by zHPF. Figure 3-6 shows an example of the optimization by a zHPF transport
mode read operation.
Status (CE/DE)
Figure 3-6 High-performance FICON read operation
The channel sends all the required CCWs and read operations of 4 KB of data in one single
frame to the CU. The CU transfers the requested data over the link to the channel, followed by
a CE/DE if the operation was successful. Less overhead is generated compared with the
existing FICON architecture.
The channel sends all the required CCWs and write operations of 4 KB of data in one frame
to the CU. The CU responds with XFER when it is ready to receive the data. The channel
then sends the 16 KB of data to the CU. If the CU successfully receives the data and finishes
the write operation, the CE/DE status is sent by the CU to indicate the completion of the write
operation.
zHPF supports multi-track operations. It allows the channel to operate at rates that fully use
the bandwidth of a FICON Express channel. The zHPF fully supports multiple tracks of data
that can be transferred in a single operation.
For more information about the FICON channel protocol and zHPF, see FICON Planning and
Implementation Guide, SG24-6497.
Information about the channels that are connected to a fabric, if they are registered, allow
other nodes or SAN managers to query the name server to determine what is connected to
the fabric. The following attributes are registered for the IBM Z platform:
Platform information:
– Worldwide node name (WWNN). This name is the node name of the platform and it is
the same for all channels that belong to the platform.
– Platform type.
– Host computer type.
– Platform name. The platform name includes vendor ID, product ID, and vendor-specific
data from the node descriptor.
Channel information.
WWPN.
The platform and name server registration service are defined in the Fibre Channel - Generic
Services 4 (FC-GS-4) standard.
In addition, FICON channels can multiplex data transfer for several devices concurrently. This
feature also allows workloads with low to moderate CU cache hit ratios to achieve higher
levels of activity rates per channel.
If the open exchange limit is reached, more I/O operations are refused by the channel, which
can result in queues and retries by the operating system.
Extended distances
Degradation of performance at extended distances can be avoided by implementing an
enhancement to the industry standard FICON architecture (FC-SB-3). This enhancement is a
protocol for persistent IU pacing. CUs that use the architecture can increase the pace count,
which is the number of IUs that are allowed to be underway between the channel and the CU.
Extended distance FICON channels retrieve the last pacing information and use this
information for later operations. This feature avoids performance degradation at the start of a
new operation.
IU pacing helps to optimize the link usage and simplifies the requirements for channel
extension equipment because more commands can be in-flight. Extended distance FICON is
apparent to the operating systems and it is applicable to all FICON features that are defined
with CHPID type FC.
CCW with IDAW flag set (with ORB specifying 4K blocks) IDAWs can start at
any address within
CCWs cmd 04 IDAW address 4K block
The usage of MIDAWs is indicated by the MIDAW bit (flag bit 7) in the CCW. The last MIDAW
in the list has a last flag set, which indicated by L in Figure 3-8. MIDAW provides
performance benefits, especially when processing extended format data sets with FICON
channels.
FICON link
The transmission medium for the FICON interface is a fiber optic cable. Physically, it is a pair
of optical fibers that provide two dedicated, unidirectional, and serial-bit transmission lines.
Information in a single optical fiber flows, bit by bit, in one direction. At any link interface, one
optical fiber is used to receive data, and the other is used to transmit data. Full-duplex
capabilities are used for data transfer. The Fibre Channel Standard (FCS) protocol specifies
that for normal I/O operations frames flow serially in both directions, allowing several
concurrent read/write I/O operations on the same link.
In general, the FICON channels and the FICON Director or CU communicate and agree on
either a 2, 4, 8,16, or 32 Gbps (that is, 200 MBps, 400 MBps, 800 MBps, 1600 MBps, or
3200 MBps) link speed. This speed determination is based on the supported speeds in the
system features, FICON Director, and CU.
Note: The link speed is the theoretical maximum unidirectional bandwidth capability of the
fiber optic link. The link data rate, whether it is measured in I/O operations per second or
MBps, depends on the type of workload, fiber infrastructure, and storage devices in place.
FICON LX features use LX transceivers and 9 µm SM fiber optic media (cables and trunks)
for data transmission. FICON SX features use SX transceivers and 50 or 62.5 µm MM fiber
optic media (cables and trunks) for data transmission. A transceiver is a transmitter and
receiver. The transmitter converts electrical signals to optical signals to be sent over the fiber
optic media. The receiver converts optical signals to electrical signals to be sent through the
system, director, or CU.
FICON
Channel FIC
ON s FICON
Fra me
(FC) me N Fra CU & I/O
s O
FIC
FC
FIC
ON switch s
Fra me
me
s N Fra
O
FIC
FC links FC links FICON
FIC
es ON Adapter
ram Fra
ONF me
FIC s
FIC
ON
FICON es F_Port Fra
ram me FICON
Channel ONF s
FIC CU & I/O
(FC)
Figure 3-9 FICON switch function
Site A Site B
Servers
Servers
FICON
Channels
Cascaded FICON Directors
FC Fabric
FICON
Channels
Switch Switch
Storage Storage
Devices Device
Generally, organizations that have data centers that are separated between two sites can
reduce the number of cross-site connections by using cascaded directors. Specific cost
savings vary depending on the infrastructure, workloads, and size of data transfers. Further
savings can be realized by reducing the number of channels and switch ports. Another
important feature of the FICON support of cascaded directors is its ability to provide
high-integrity data paths. The high-integrity function is an integral component of the FICON
architecture when you are configuring FICON channel paths through a cascaded fabric.
To support the introduction of FICON cascaded switching, IBM worked with Fibre Channel
Director vendors. IBM and the vendors worked to ensure that robustness in the
channel-to-CU path is maintained to the same high standard of error detection, recovery, and
data integrity as the initial implementation of FICON.
A FICON channel, when configured to operate with a cascaded switch fabric, requires that the
switch fabric supports high integrity. During initialization, the FICON channel queries the
switch fabric to determine whether it supports high integrity. If it does, the channel completes
the initialization process, allowing the channel to operate with the fabric.
After a FICON switched fabric is customized to support FICON cascaded directors and the
required WWNN and domain IDs are added in the fabric membership list, the director checks
that its ISLs are attached to the correct director before they are operational. If an accidental
cable swap occurs, the director starts logical path testing, reporting, isolation, and recovery.
The high-integrity fabric feature for cascaded FICON Directors protects against miscabling
and misdirecting of data streams, as shown in Figure 3-11.
FICON 1 FICON
6
Director Director
4v !
FICON FICON
Director Director
6
A static SAN routing policy typically assigns the ISL routes according to the incoming port and
its destination domain (port-based routing (PBR), or the source and destination ports pairing
(device-based routing (DBR)).
PBR assigns the ISL routes statically based on first come, first served when a port starts a
fabric login (FLOGI) to a destination domain. The ISL is round-robin for assignment. Thus, I/O
flow from the same incoming port to the same destination domain is always assigned the
same ISL route, regardless of the destination port of each I/O. This process can result in
some ISLs being overloaded while others are underutilized. The ISL routing table is changed
every time that a IBM Z server undergoes a power-on-reset (POR). Because of this POR, the
ISL assignment is unpredictable.
DBR assigns the ISL routes statically based on a hash of the source and destination port.
That I/O flow from the same incoming port to the same destination is assigned with the same
ISL route. Compared to PBR, DBR can better spread load across ISLs for I/O flow from the
same incoming port to different destination ports within the same destination domain.
When using a static SAN routing policy, the FICON director has limited capability to assign
ISL routes based on workload. There are also chances of ISL overloaded or underutilization.
With dynamic routing, ISL routes are dynamically changed based on the FC exchange ID,
which is unique for each I/O operation. The ISL is assigned at I/O request time so that
different I/Os from the same incoming port to the same destination port are assigned different
ISLs.
ISL assigned at
I/O request time
Dynamic routing (Brocade EBR or Cisco OxID) dynamically changes the routing between the
channel and CU based on the FC Exchange ID. Each I/O operation has a unique exchange
ID. Here are some of the benefits of FIDR:
Reduces cost by allowing sharing of ISLs between FICON and FCP (PPRC or distributed).
Provides better balanced I/O traffic between all available ISLs.
Improves utilization of switch and ISL hardware.
Provides easier management.
Allows for easier capacity planning for ISL bandwidth requirements.
Provides predictable, repeatable I/O performance.
FICON does not allow multiple physical CUs, or CU link interfaces, to be on the same link. A
CU can contain multiple images with dedicated or shared physical CU facilities. The FICON
I/O architecture provides addressing for these multiple CU images.
FICON channels can be spanned across multiple CSSs in the IBM Z platform. For more
information about MIF and spanned channels, see 2.1.2, “Multiple CSSs” on page 19.
2
Applies to the FICON Express32S, FICON Express16SA, FICON Express16S+, FICON Express16S, and FICON
Express8S features that are supported by z/OS, z/VM, and Linux on IBM Z.
IBM Tivoli® System Automation for z/OS (SA for z/OS) includes support for FICON channels
and FICON Directors. You can find more information, updates, extensions, tools, and
demonstrations at the IBM DevOps web page.
Before using SA for z/OS in your FICON environment, check the latest maintenance
recommendations in the appropriate z/OS subset of the appropriate PSP buckets
(3931DEVICE, 8561DEVICE, 3906DEVICE, 3932DEVICE, 8562DEVICE, or 3907DEVICE)
before implementation.
The zDAC function is integrated into the existing HCD tool. A policy can be defined in the
HCD according to the availability and bandwidth requirements, including parallel access
volume (PAV) definitions, CU numbers, and device number ranges. The zDAC proposed
configurations are created as work input/output definition files (IODFs) that can be converted
to a production IODF and activated.
zDAC provides real-time discovery for the FICON fabric, subsystem, and I/O device resource
changes from z/OS. By exploring the discovered CUs for defined LCUs and devices, zDAC
compares the discovered controller information with the current system configuration to
determine delta changes to the configuration for a proposed configuration.
All new added or changed logical CUs and devices are added to the proposed configuration
with proposed CU and device numbers and channel paths based on the defined policy. zDAC
uses a channel path-chosen algorithm to minimize single points of failure. zDAC applies to all
FICON features that are supported on the IBM Z platform when configured as CHPID type
FC.
Table 3-2 lists the available FICON features and their respective specifications. All FICON
features use LC duplex connectors. For LX FICON features that can use a data rate of
1 Gbps, mode-conditioning patch (MCP) cables, either 50 or 62.5 MM, can be used. The
maximum distance for this connection is reduced to 550 m at a link data rate of 1 Gbps.
Details for each feature follow the table.
The ports on a single FICON Express16S and FICON Express8S feature can be configured
individually and defined in different channel modes (FC and FCP).
The FICON Express32S features have two independent ports. Each feature occupies a single
I/O slot by using one CHPID per channel. Each channel supports 8 Gbps, 16 Gbps, or
32 Gbps link data rates with auto-negotiation. The FICON Express32S is supported on
IBM z16.
FICON Express32S helps absorb large application and transaction spikes that are driven by
large unpredictable AI and hybrid cloud workloads, and it is a key component of the IBM Fibre
Channel Endpoint Security solution.
Each FICON Express32S port has a PCHID and can be defined as CHPID type FC or FCP.
However, both ports must be the same CHPID type. FICON Express32S CHPIDs can be
defined as a spanned channel and shared among LPARs within and across CSSs.
The FICON Express32S features are installed in the Peripheral Component Interconnect
Express+ (PCIe+) I/O drawer and use SFP optics to permit each channel to be individually
serviced during a fiber optic module failure. Traffic on the other channels on the same feature
can continue to flow if a channel requires servicing.
The FICON Express32S features are ordered in two channel increments and are added
concurrently. This concurrent update capability allows you to continue to run workloads
through other channels when the FICON Express32S features are being added.
The FICON Express32S features are designed for connectivity to systems, switches,
directors, disks, tapes, and printers, and can be defined in two ways:
CHPID type FC:
– Native FICON, zHPF, and FICON CTC traffic
– Supported in z/OS, IBM z/VM, IBM z/VSE V6.2 with program temporary fixes (PTFs),
IBM z/TPF V1.1 with PTFs, and Linux on IBM Z3
CHPID type FCP:
– FCP traffic for communication with SCSI devices
– Supported in IBM z/VM, z/VSE V6.2 with PTFs, and Linux on IBM Z
3
The support statements for Linux on IBM Z server also cover the KVM hypervisor on distribution levels that have
KVM support.
The FICON Express16SA features can be placed only in the PCIe+ I/O drawer, and they use
SFP optics to permit each channel to be individually serviced during a fiber optic module
failure. Traffic on the other channels on the same feature can continue to flow if a channel
requires servicing.
The FICON Express16SA features are ordered in two channel increments and are added
concurrently. This concurrent update capability allows you to continue to run workloads
through other channels when the FICON Express16SA features are being added.
The FICON Express16SA features are designed for connectivity to systems, switches,
directors, disks, tapes, and printers, and can be defined in two ways:
CHPID type FC:
– Native FICON, zHPF, and FICON CTC traffic
– Supported in the z/OS, IBM z/VM V7.1 with corresponding APARs, IBM z/VSE V6.2
with PTFs, IBM z/TPF V1.1 with PTFs, and Linux on IBM Z4
CHPID type FCP:
– FCP traffic for communication with SCSI devices
– Supported in IBM z/VM V6.4 and V7.1, z/VSE V6.2 with PTFs, and Linux on IBM Z
An IBM Security Key Lifecycle Manager server provides shared secret key generation in a
master and subordinate relationship between an FC initiator (IBM Z) and the storage target.
The solution implements authentication and key management features through IBM Secure
Key Exchange (SKE).
Data that is in-flight (from or to IBM Z and IBM Storage) is encrypted when it leaves either
endpoint (source) and decrypted at the destination. Encryption and decryption are done at
the FC adapter level. The operating system that is running on the host (IBM Z) is not involved
in IBM Fibre Channel Endpoint Security related operations. Tools are provided at the
operating system level for displaying information about encryption status.
IBM Fibre Channel Endpoint Security is an orderable feature (Feature Code 1146) for
IBM z16 and IBM z15 T015, and it requires Central Processor Assist for Cryptographic
Function (CPACF) enablement (Feature Code 3863), specific storage (DS8900), and FICON
Express32S or FICON Express16SA features.
4
The support statements for Linux on IBM Z server also cover the KVM hypervisor on distribution levels that have
KVM support.
5 Not supported by IBM z15 T02.
Each FICON Express16S+ feature has two ports. Each port has a PCHID and can be defined
as an FC or FCP type. However, both ports must be the same CHPID type. FICON
Express16S+ CHPIDs can be defined as a spanned channel and shared among LPARs
within and across CSSs.
All FICON Express16S+ features are in the Peripheral Component Interconnect Express
(PCIe) I/O drawer or PCIe+ I/O drawer6 and use SFP optics to permit each channel to be
individually serviced during a fiber optic module failure. Traffic on the other channels on the
same feature can continue to flow if a channel requires servicing.
The FICON Express16S+ features are ordered in two channel increments and added
concurrently. This concurrent update capability allows you to continue to run workloads
through other channels when the FICON Express16S+ features are being added.
The FICON Express16S+ features are designed for connectivity to systems, switches,
directors, disks, tapes, and printers, and can be defined in two ways:
CHPID type FC:
– Native FICON, zHPF, and FICON CTC traffic
– Supported in the z/OS, IBM z/VM hypervisor, IBM z/VSE V6.2 (earlier z/VSE versions
have no zHPF support), IBM z/Transaction Processing Facility (z/TPF), and Linux on
IBM Z and the KVM hypervisor environments
CHPID type FCP:
– FCP traffic for communication with SCSI devices
– Supported in IBM z/VM, z/VSE, and Linux on IBM Z and the KVM hypervisor
environments
6 PCIe+ I/O drawer (Feature Code 4023 on IBM z16, Feature Code 4021 on IBM z15, and Feature Code 4001 on
IBM z14 ZR1) replaces the PCIe I/O drawer. All PCIe features that can be carried forward during an upgrade can
be installed in the PCIe+ I/O drawer. PCIe I/O drawer (Feature Code 4032) cannot be carried forward during an
upgrade.
All FICON Express16S features are in the PCIe I/O drawer or PCIe+ I/O drawer, and they use
SFP optics to permit each channel to be individually serviced during a fiber optic module
failure. Traffic on the other channels on the same feature can continue to flow if a channel
requires servicing.
The FICON Express16S features are ordered in two channel increments and added
concurrently. This concurrent update capability allows you to continue to run workloads
through other channels when the FICON Express16S features are being added.
FICON Express16S CHPIDs can be defined as a spanned channel and shared among
LPARs within and across CSSs.
The FICON Express16S features are designed for connectivity to systems, switches,
directors, disks, tapes, and printers, and can be defined in two ways:
CHPID type FC:
– Native FICON, zHPF, and FICON CTC traffic
– Supported in the z/OS, IBM z/VM hypervisor, IBM z/VSE V6.2 (earlier z/VSE versions
have no zHPF support), IBM z/TPF, and Linux on IBM Z and the KVM hypervisor
environments
CHPID type FCP:
– FCP traffic for communication with SCSI devices
– Supported in IBM z/VM, z/VSE, and Linux on IBM Z and the KVM hypervisor
environments
These features are supported on the IBM Z platform (carry forward only on IBM z15, IBM z14,
and IBM z14 ZR1), and are designed to deliver increased performance in comparison to
FICON Express8 features. The FICON Express8S features include half the number of ports
per feature in comparison with the FICON Express8 features. This design facilitates
purchasing the correct number of ports to help satisfy your application requirements and
better optimize for redundancy.
All FICON Express8S features are in the PCIe I/O drawer or PCIe+ I/O drawer and use SFP
optics to permit each channel to be individually serviced during a fiber optic module failure.
Traffic on the other channels on the same feature can continue to flow if a channel requires
servicing.
The FICON Express8S features are ordered in two channel increments and are added
concurrently. This concurrent update capability allows you to continue to run workloads
through other channels when the FICON Express8S features are being added.
FICON Express8S CHPIDs can be defined as a spanned channel and shared among LPARs
within and across CSSs.
On the left side of the web page, select Library and locate the listing for “Hardware products
for servers” in the middle of the web page. Then, select Switches and directors that are
qualified for IBM Z FICON and FCP channels.
Note: Certain functions might require specific levels of an operating system, PTFs, or both.
That information is provided when necessary within this chapter.
With these reports, you can analyze possible bandwidth bottlenecks to determine the cause.
For more information about performance, see the IBM Z I/O connectivity web page.
For more information about FCS publications, see the Technical Committee T11 website.
For more information about the SCSI Storage Interface standards, see the Technical
Committee T10 website.
This chapter describes the zHyperLink connectivity option, which is offered with the IBM z16,
IBM z15, and IBM z14, with the IBM Storage DS888x and DS8900.
Note: The zHyperLink Express1.1 (Feature Code 0451) is a technology refresh that
includes the same functional and software requirements as the zHyperLink Express
(Feature Code 0431). Throughout this chapter, both features are referred to as zHyperLink
or zHyperLink Express, unless otherwise specified.
zHyperLink Express is designed for up to 5X lower read latency than high-performance Fibre
Channel connection (FICON).1 Working with the FICON storage area network Infrastructure,
zHyperLink can improve application response time by cutting I/O-sensitive workload response
time by up to 50% without requiring application changes.
The zHyperLink Express feature in the IBM z16, IBM z15, IBM z14, and IBM z14 ZR1 allows
you to make Synchronous I/O requests for data that is in the cache of the IBM DS888x or
newer model storage. This task is done by directly connecting a zHyperLink Express port in
the IBM z16, IBM z15, IBM z14, or IBM z14 ZR1 to an I/O Bay zHyperLink port of the DS888x
(or newer).
To better plan your zHyperLink implementation, use the PC-based tool IBM Z Batch Network
Analyzer (IBM zBNA), which provides graphical and text reports. For more information, see
this IBM Support web page.
Note: zHyperLink connections work with FICON channels; they do not replace them (a
FICON channel is required to “drive” the zHyperLink). Only z/OS and Extended Count Key
Data (ECKD) are supported, and the z/OS image must run in a logical partition (LPAR), not
as a guest under IBM z/VM.
A new Synchronous I/O command was defined to z/OS to synchronously read one or more
data records. In addition, an option is provided to allow z/OS to initiate a Synchronous I/O
command and return control to perform other processing requests.
When a traditional I/O operation is requested by a task to start the Input/Output Supervisor
(IOS), the I/O operation is not handled by the central processor that is assigned to the z/OS
LPAR. There are specialized components to do the job: system assist processors (SAPs) and
channel programs.
An SAP is a special processing unit (PU) that helps set up the I/O operation. The SAP finds
an available channel path, but is not responsible for the data movement between the storage
control unit (CU) and z/OS LPAR memory. The channel program communicates with CU to
manage the data movement. After the I/O operation completes, an I/O interrupt notifies the
central processor so that IOS can be run again.
1 Results observed in an IBM internal lab. Actual performance can vary depending on the workload.
4.3 Connectivity
IBM zHyperLink Express takes up one slot in the Peripheral Component Interconnect
Express (PCIe) I/O drawer or the PCIe+ I/O drawer, and it has two ports. Both ports are on a
single physical channel ID (PCHID). The zHyperLink Express uses PCIe Gen3 technology,
with x16 lanes that are bifurcated into x8 lanes. It is designed to support distances up to
150 m at a link data rate of 8 gigabits per second (Gbps).
Figure 4-1 shows a zHyperLink Express feature in an IBM z15 connecting to a zHyperLink
feature in the I/O Bay of a DS888x (or later). This configuration is valid for the IBM z14 and
IBM z16.
The zHyperLink Express requires 24x Multi-fiber Termination Push-on (MTP)-MTP cables to
connect to a DS888x (or newer) and z/OS 2.1 or later.
Up to 16 zHyperLink Express adapters can be installed in an IBM z16, IBM z15, and IBM z14
for up to 32 links.
The zHyperLink Express feature works as a native PCIe adapter that can be shared by
multiple LPARs. Each port can support up to 127 virtual functions (VFs), with one or more
VFs or PFIDs being assigned to each LPAR. This configuration supports a maximum of 254
VFs per adapter.
At a minimum, a DS888x with firmware R8.3 is required, with the I/O Bay system board
updated to support the zHyperLink interface.
Two fiber options are available with specifications that support different distances for the
zHyperLink Express features:
Up to 150 m: OM4 50/125 micrometer multimode (MM) fiber optic cable with a fiber
bandwidth wavelength of 4.7 GHz-km @ 850 nm.
Up to 100 m: OM3 50/125 micrometer MM fiber optic cable with a fiber bandwidth
wavelength of 2.0 GHz-km @ 850 nm.
4.4 References
The following publications contain information that is related to the topics that are covered in
this chapter:
Getting Started with IBM zHyperLink for z/OS, REDP-5493
IBM DS8900F and IBM Z Synergy DS8900F: Release 9.3 and z/OS 2.5, REDP-5186
IBM z16 (3931) Technical Guide, SG24-8951
IBM z16 A02 and IBM z16 AGZ Technical Guide, SG24-8952
Terminology: The terms OSA and OSA-Express are used throughout this chapter to
simplify descriptions and discussion that are related to common functions and support
across all OSA-Express features.
Table 5-1 provides an overview of the types of traffic that are supported by IBM Z platforms for
each CHPID type. It also indicates whether the Open Systems Adapter Support Facility
(OSA/SF) is required to configure the OSA-Express ports that are based on the supported
modes of operation (CHPID types).
1 IBM z16 is planned to be the last IBM Z Server to support OSE networking channels. IBM Z support for the System
Network Architecture (SNA) protocol being transported natively out of the server by using OSA-Express
100BASE-T adapters configured as channel type “OSE” will be eliminated after IBM z16.
2 For more information, see Chapter 6, “Console Communications (OSA-ICC)” on page 129.
The OSA-Express 1000BASE-T features support non-QDIO mode. This mode supports SNA,
Advanced Peer-to-Peer Networking (APPN), high-performance routing (HPR), and IP network
traffic simultaneously through the OSA port. This section describes the non-QDIO mode
types.
IP network pass-through
In IP network pass-through mode, an OSA feature transfers data between an IP network
stack to which it is defined and clients on an Ethernet 100/10003 Mbps LAN. The LAN is
attached to the port on a 1000BASE-T feature and supports one of the following frame
protocols:
Ethernet II that uses the DEC Ethernet V 2.0 envelope
Ethernet 802.3 that uses the 802.2 envelope with SNAP
For IP network pass-through mode, the default OSA Address Table (OAT) can be used. In that
case, no configuration or setup is required.
OSA-Express features use Direct Memory Access (DMA) and a data router model to
eliminate store and forward delays.
Also in QDIO mode, all OSA features dynamically receive configuration information from the
host. This process reduces configuration and setup time, eliminates duplicate data entry, and
reduces the possibility of data entry errors and incompatible definitions.
QDIO is the interface between the operating system and the OSA hardware.
3 OSA-Express7S and newer support only 1000 Mbps (no negotiation at lower speeds).
For more information about the Layer 2 support, see 5.2.13, “Layer 2 support” on page 87.
Data router
The IBM Application Specific Integrated Circuit (ASIC) processor of the OSA feature handles
packet construction, inspection, and routing. This feature allows packets to flow between host
memory and the LAN at line speed without firmware intervention. With the data router, the
store and forward technique in DMA is no longer used, which enables a direct host
memory-to-LAN flow and avoids a hop. It reduces latency and increases throughput for
standard frames (1492 bytes) and jumbo frames (8992 bytes).
Priority queuing
Priority queuing is supported by the QDIO architecture. It was introduced with the Service
Policy Server (for z/OS environments only). It sorts outgoing IP message traffic according to
the service policy that is set up for the specific priority that is assigned in the IP header.
This capability is an alternative to the best-effort priority that is assigned to all traffic in most
IP networks. Priority queuing allows the definition of four different priority levels for IP network
traffic through the OSA features that are defined for QDIO. For example, the highest priority
can be granted to interactive communications, and the lowest priority can be granted to batch
traffic, with two more categories in between based on user groups or projects.
QDIO uses four write (outbound) queues and one read (inbound) queue for each IP network
stack that is sharing the OSA feature.
OSA signals the z/OS Communications Server when there is work to do. z/OS
Communications Server puts outbound packets in one of the four queues, based on priority
settings.
At a certain time, z/OS Communications Server signals the OSA feature that there is work to
do. The OSA feature searches the four possible outbound queues by priority and sends the
packets to the network, giving more priority to queues 1 and 2, and less priority to queues 3
and 4. For example, if there is data on every queue, queue 1 is served first, then portions of
queue 2, then fewer portions of queue 3, then even fewer portions of queue 4, and then back
to queue 1. This process means that four transactions are running across the four queues.
Over time, queue 1 finishes first, queue 2 finishes second, and so on.
Note: With OSA-Express, priority queuing is enabled by default. This feature reduces the
total number of supported IP network stacks and devices. For more information, see
“Maximum IP network stacks and subchannels” on page 78.
The OAT entries are dynamically built when the corresponding IP device in the IP network
stack is started.
At device activation, all IP addresses that are contained in the IP network stack’s IP HOME
list are downloaded to the OSA port. Corresponding entries are built in the OAT. Subsequent
changes to these IP addresses cause an update of the OAT.
LPAR-to-LPAR communication
Access to an OSA port can be shared among the system images that are running in the
LPARs to which the channel path is defined to be shared. Also, access to a port can be
shared concurrently among IP network stacks in the same LPAR or in different LPARs.
When sharing ports, an OSA port operating in QDIO mode can send and receive IP traffic
between LPARs without sending the IP packets to the LAN and then back to the destination
LPAR.
For outbound IP packets, the OSA port uses the next-hop IP address within the packet to
determine where it is sent. If the next-hop IP address was registered by another IP network
stack that is sharing the OSA port, the packet is sent directly to that IP network stack, not onto
the LAN. This feature makes the forwarding of IP packets possible within the same host
system.
QDIO functions
The QDIO functions that are described in this section are supported on the IBM Z platform.
IP network functions
The following IP network functions are available:
Large send for IP network traffic for OSA-Express (for more information, see 5.2.4, “Large
send for IP network traffic” on page 80)
640 IP network stacks (for more information, see “Maximum IP network stacks and
subchannels” on page 78)
These hardware assists allow a cooperating guest operating system to start QDIO operations
directly to the applicable channel without interception by z/VM, which helps improve
performance. Support is integrated into IBM Z Licensed Internal Code (LIC).
This process allows the host operating system to signal the OSA-Express features to stop
traces and capture the current trace records. By using existing tools (traps) and commands,
the operator can capture both hardware and software traces at the same time and then
correlate the records during post-processing.
When the OSA port is defined in QDIO mode, the OAT entries are built and updated
dynamically.
The MAC or VMAC addresses are added to the Layer 2 table of the OAT when the IP network
stacks (in which the addresses are defined) are started.
The following maximum number of IP network stacks and subchannels are supported:
OSA port in non-QDIO mode (CHPID type OSE)
An OSA port in non-QDIO mode can support up to 120 IP network stacks and
240 subchannels for all IBM Z platforms.
OSA port in QDIO mode (CHPID type OSD)
The OSA features support 640 IP network stack connections per dedicated CHPID, or
640 total stacks across multiple LPARs when defined as a shared or spanned CHPID. The
maximum number of subchannels that are allowed is 1920 (1920 subchannels / 3 =
640 stacks).
Note: By default, OSA-Express features have multiple priorities for outbound queues
enabled (four QDIO priorities). Therefore, the maximum number of supported subchannels
is reduced to 480 (1920 subchannels / 4 = 480 subchannels), which reduces the total
number of supported IP network stacks to 160 (480 subchannels / 3 = 160 stacks). Priority
queues can be disabled through HCD or IOCP. For example, in IOCP, use the CHPARM=02
value to disable priority queuing.
Usage of the GUI is optional. A REXX CLI is also included with OSA/SF. OSA/SF is integrated
into z/OS, z/VM, and z/VSE, and runs as a host application. For OSA/SF, Java GUI
communication is supported through IP networks only. This version of OSA/SF is not offered
as a separately licensed product.
The HMC is enhanced to use the OSA/SF function for the OSA-Express features. OSA/SF on
the HMC or OSA/SF in the operating system component can be used for the OSA-Express
features (6S, 7S, and 7S 1.2). For the OSA-Express7S 1.2, OSA-Express7S, and
OSA-Express6S features, OSA/SF on the HMC is required.
OSA/SF is not always required to customize an OSA feature. However, it can be used to
gather operational information to help with problem determination to monitor and control
ports. The OSA/SF Query function provides performance information about the OSA
CHPIDs. As shown in Table 5-1 on page 72, OSA/SF is not required to configure the OSA
features in any operating modes except OSE.
An IP address traditionally ties to a physical link at one end of a connection. If the associated
physical link goes down, it is unreachable. The VIPA exists only in software and has no
association to any physical link. The IP network stack is the destination IP address rather than
the network attachment.
For an OSA port to forward IP packets to a particular IP network stack for routing to its
destination, the PRIRouter must be defined on the DEVICE statement in the IP network profile.
If the IP network stack that has an OSA port that is defined as PRIRouter becomes
unavailable, the following process occurs: A second IP network stack that is defined as the
secondary router (SECRouter on the DEVICE statement in the IP network profile) receives the
packets for unknown IP addresses.
Important Sharing a single OSA port (with a single MAC address) can fail in
load-balancing solutions. A workaround is to use Generic Routing Encapsulation (GRE) or
network address translation (NAT), which can have a negative effect on performance. A
Layer 3 VMAC is a function that is available on IBM Z platforms with OSA features that
allows multiple MAC addresses on a single OSA port. For more information, see 5.2.16,
“Layer 3 VMAC for z/OS” on page 91.
Large send support reduces host processor use, so it returns processor cycles for application
use and increases network efficiencies. For OSA-Express features, large send supports
outbound IPv4 traffic only and applies solely to unicasts. Large send support for IPv6 packets
applies to the OSA-Express features (CHPID types OSD) that are available on the IBM Z
platform.
Note: Large send for IPv6 packets is not supported for LPAR-to-LPAR packets.
The IEEE standard 802.1q describes the operation of virtual bridged local area networks. A
VLAN is defined as a subset of the active topology of a local area network. The OSA features
provide for the setting of multiple unique VLAN IDs per QDIO data device. They also provide
for both tagged and untagged frames to flow from an OSA port. The number of VLANs that is
supported is specific to the operating system.
4
For more information about the KVM hypervisor, see:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liaaf/lnz_r_kvm.html
With GVRP support, an OSA-Express port can register or unregister its VLAN IDs with a
GVRP-capable switch and dynamically update its table as the VLANs change. Support of
GVRP is exclusive to IBM Z. It is applicable to all the OSA-Express features when in QDIO
mode (CHPID type OSD), and it is supported by z/OS and z/VM.
5.2.6 Simple Network Management Protocol support for z/OS and Linux on
IBM Z
Simple Network Management Protocol (SNMP) is supported for OSA features when
configured in the QDIO mode (CHPID type OSD). The OSA features LIC includes the
following support for the OSA SNMP subagents:
Get and GetNext requests
This support applies to all OSA features that are supported on the IBM Z platform.
dot3StatsTable
Ethernet data for dot3StatsTable applies to all Ethernet features that are supported on IBM
Z platform. It implements the SNMP EtherLike Management Information Base (MIB)
module of RFC 2665, which provides statistics for Ethernet interfaces. These statistics can
help in the analysis of network traffic congestion.
Performance data
This support applies to all OSA features that are supported on the IBM Z platform. The
performance data reflects the OSA usage.
Traps and Set
This support applies to all OSA features that are supported on IBM Z.
SNMP support for LAN channel station (LCS) applies to all OSA features that are supported
on IBM Z, along with IP network applications only. It supports the same SNMP requests and
alerts that are offered in QDIO mode (Get, GetNext, Trap, and Set), and it is exclusive to the
z/OS environment.
Tip: You can subscribe to the OSA-Express Direct SNMP MIB module document through
IBM Resource Link to receive email notifications of document changes.
OSA/SF is not required to manage SNMP data for the OSA features. An SNMP subagent
exists on an OSA feature, which is part of a direct path between the z/OS or Linux on IBM Z
master agent (IP network stacks) and an OSA-Express MIB.
Multicast support
For sending data to multiple recipients, OSA features support IP multicast destinations only in
QDIO or IP pass-through mode.
A broadcast simultaneously transmits data to more than one destination. Messages are
transmitted to all stations in a network (for example, a warning message from a system
operator). The broadcast frames can be propagated through an OSA feature to all IP network
applications that require broadcast support, including applications that use RIP V1.
Purging ARP entries in cache for IPv4 for z/OS and Linux on IBM Z
Purging entries in the ARP cache is supported by using IPv4. The IP network stack already
has an awareness of IPv6 addresses.
ARP takeover
ARP takeover provides the capability of switching OSA port operations from one OSA to
another OSA that are running in the same mode in z/OS environments.
When a z/OS IP network is started in QDIO mode, it downloads all home IP addresses in the
stack and stores them in each OSA feature to which it has a connection. This service is part
of the QDIO architecture and occurs automatically only for OSD channels. For OSA ports that
are set up as OSE channels (non-QDIO), multiple IP addresses must be defined in the OAT
by using OSA/SF. Then, OSA responds to ARP requests for its own IP address and for VIPAs.
If an OSA feature fails and a backup OSA is available on the same network or subnetwork,
the IP network informs the backup OSA which IP addresses (real and VIPA) to take over, and
the network connection is maintained. For this technique to work, multiple paths must be
defined to the IP network stack. For example, MULTIPATH must be defined to the IPCONFIG
statement of the IP network profile in z/OS.
ARP statistics
QDIO includes an IPA function, which gathers ARP data during the mapping of IP addresses
to MAC addresses. CHPIDs that are defined as OSD maintain ARP cache information in the
OSA feature (ARP offload). This data is useful in problem determination for the OSA feature.
Checksum offload is supported by all OSA Ethernet features when operating in QDIO mode.
By offloading checksum processing to the supporting OSA features, host system cycles are
reduced, which can result in improved performance for most IPv4 packets.
Note: Linux on IBM Z supports only inbound checksum offload (inbound packets).
When the checksum is offloaded, the OSA feature calculates the checksum. When multiple IP
stacks share an OSA port and an IP stack sends a packet to a next-hop IP address that is
owned by another IP stack that shares an OSA port, OSA sends the IP packet directly to the
other IP stack without placing it on the LAN. Checksum offload does not apply to such IP
packets.
The checksum offload is performed for IPv6 packets and IPv4 packets. This process occurs
regardless of whether the traffic goes to the LAN, comes in from the LAN, or flows from one
LPAR to another LPAR through the OSA feature.
Checksum offload for IPv6 packets is supported by CHPID type OSD on the IBM Z platform.
Checksum offload for LPAR-to-LPAR traffic in the z/OS environment is included in the
OSA-Express design for both IPv4 and IPv6 packets.
Checksum offload support is available with z/OS and z/VM. For more information, see 5.3.3,
“Software support” on page 103.
Dynamic LAN idle reduces latency and improves networking performance by dynamically
adjusting the inbound blocking algorithm. When enabled, the z/OS IP network stack adjusts
the inbound blocking algorithm to best match the application requirements.
System administrators can authorize the z/OS IP network stack to enable a dynamic setting,
which was previously a static setting. The z/OS IP network stack dynamically determines the
best setting for the current running application, which is based on system configuration,
system, inbound workload volume, processor use, traffic patterns, and related items.
OLM is supported by the Communications Server for z/OS with any OSA features on the
IBM Z platform.
The Layer 2 transport mode allows for communication with IP and non-IP protocols. OSA
works with either a z/VM IP network or Linux on IBM Z Layer 2 support that is running in an
LPAR or as a z/VM guest.
The z/VM virtual switch can also be used to enable the Layer 2 function for guest systems
(see Figure 5-6).
The virtual switch uses each guest system’s unique MAC address to forward frames. Data is
transported and delivered within Ethernet frames. This process transports both IP and non-IP
frames (for example, NetBIOS and SNA) through the fabric that the virtual switch supports.
Through the address-resolution process, each guest system’s MAC address becomes known
to hosts on the physical side of the LAN segment. All inbound or outbound frames that pass
through the OSA port have the guest system’s corresponding MAC address as the source or
destination address.
The OSA Ethernet features can filter inbound frames by virtual local area network
identification (VLAN ID, IEEE 802.1q), the Ethernet destination MAC address, or both.
Filtering can reduce the amount of inbound traffic that is being processed by the operating
system, which reduces processor use. Filtering by VLAN ID or MAC address can also enable
you to isolate portions of your environment that have sensitive data to provide a degree of
low-level security.
Note: Target links for aggregation must be of the same type (for example, GbE to GbE).
Two modes allow for connection isolation: Port isolation and Virtual Ethernet Port
Aggregator (VEPA). Port isolation and VEPA are mutually exclusive.
An isolated virtual switch cannot communicate directly with other LPARs that share the
OSA-Express port. Communications must be relayed through another network-based device,
such as a router or firewall, in the external network.
As shown in Figure 5-7, when in isolation mode, data traffic that is destined for a guest
system port in the VSWITCH is blocked. However, traffic that is going to an external network
device is sent to the OSA-Express port for delivery. The isolation options (ON or OFF) can be
set by using the SET VSWITCH ISOLATION command.
For a VSWITCH to enter VEPA mode, the external Ethernet switch must be in Reflective
Relay mode.
For example, as shown in Figure 5-9 on page 91, interface isolation is enabled for LPAR 1.
Therefore, data traffic from LPAR 2 that is destined for LPAR 1 is dropped by the OSA.
QDIO interface isolation is supported by the Communications Server for z/OS with OSA
features on the IBM Z platform.
OSA Layer 3 VMAC for z/OS is applicable to OSA Ethernet features when configured as
CHPID type OSD (QDIO).
EE requires APPN or HPR at the endpoints. To enable EE, you must configure the IP network
stack with a VIPA and define an XCA major node. The XCA major node is used to define the
PORT, GROUP, and LINE statements for the EE connections.
z/OS Communications Server also supports a secure, RACF-based single sign-on logic
called Express Logon Facility (ELF). ELF works with IBM TN3270 clients to securely
authenticate the client, acquire a pass token, and then pass it on to the TN3270E server for
replacement or submission to the application.
In extending the usage of adapter interruptions to OSD (QDIO) channels, the programming
overhead to process a traditional I/O interruption is reduced. This technique benefits OSA IP
network support in Linux on IBM Z, z/VM, and z/VSE.
Adapter interruptions apply to all OSA features on the IBM Z platform when in QDIO mode
(CHPID type OSD).
Figure 5-10 shows the IWQ concept of using multiple inbound queues for different types of
workloads (T1 through T4) compared to a single inbound queue.
The data from a specific workload type is placed in one of four input queues, and a process is
created and scheduled to run on one of multiple processors, independent from the three other
queues. This process greatly improves performance because IWQ can use the symmetric
multiprocessor (SMP) architecture of the IBM Z platform.
Users already using IWQ should be aware that each IWQ workload type that applies to their
z/OS environment (for example, Sysplex Distributor, EE) is automatically enabled when IWQ
is enabled. Each unique input queue consumes extra fixed ECSA memory (approximately 4
MB per input queue per OSA interface).
OSA devices
The different types of OSA channels (CHPID types) require the following device types:
OSA devices for QDIO (OSD) and non-QDIO (OSE) CHPID types.
3270-X and 3287 devices (consoles) for the OSA-ICC (OSC) CHPID type.
OSA/SF requires one device (defined through HCD) to be associated with the OSA
CHPID as device type OSAD (UNITADD=FE). OSA/SF uses this device to communicate
with the OSA feature.
The OSA-Express Network Traffic Analyzer for z/OS requires one or more data paths
devices for the OSAENTA trace interface, depending on the configuration.
Spanned channels
Spanning is the ability to configure channels to multiple channel subsystems (CSSs). When
defined that way, the channels can be transparently shared by any or all configured LPARs,
regardless of the CSS to which the LPAR is configured.
OSA ports can be spanned across multiple CSSs on the IBM Z platform. For more
information about spanned channels, see 2.1.2, “Multiple CSSs” on page 19.
OSA-Express features are offered in two transmission medium types with different speeds:
OSA-Express fiber Ethernet features use Small Form-factor Pluggable (SFP)5 transceiver
technology6 with LC duplex receptacles and require either multimode (MM) at 62.5 µm or
50 µm fiber optic cables, or single-mode (SM) at 9 µm fiber optic cables. OSA-Express
fiber Ethernet features, which are known as either GbE, 10 GbE, or 25 GbE (based on the
speed (gigabit per second) (GbE) of the feature), do not support auto-negotiate to a lower
speed.
The OSA-Express copper Ethernet features use RJ45 jacks and require unshielded
twisted pair (UTP) Cat5 or Cat6 cables. OSA-Express copper Ethernet features7 (referred
to as 1000BASE-T) support auto-negotiate at the following speeds:
– 10 Mbps half-duplex or full-duplex8
– 100 Mbps half-duplex or full-duplex
– 1000 Mbps full-duplex
5 SFP allows for a concurrent repair or replace action.
6
Except for the OSA-Express4S features.
7
OSA-Express7S 1000BASE-T feature does not support auto-negotiate to a lower speed.
8 OSA-Express6S and OSA-Express5S 1000BASE-T features do not support 10 Mbps.
LAN speed can be set explicitly by using OSA/SF or the OSA Advanced Facilities function of
the HMC. The explicit settings override the OSA feature port’s ability to auto-negotiate with its
attached Ethernet switch.
Table 5-2 lists the OSA-Express features that are supported on the respective IBM Z platform
with the maximum unrepeated distance and cable type.
OSA-Express7S GbE SX 0443 MM 62.5 µm 275 m (200) IBM z16 A01b and
MM 50 µm 550 m (500) IBM z15 T01
OSA-Express7S 1000BASE-T 0446 UTP Cat5 or 100 m IBM z16 A01b and
Cat6 IBM z15 T01
OSA-Express5S 10GbE SRc 0416 MM 50 µm 550 m (500) IBM z15 T01b and
IBM z14b
MM 62.5 µm 275 m (200)
220 m (160)
OSA-Express5S GbE SXc 0414 MM 50 µm 550 m (500) IBM z15 T01b and
IBM z14b
MM 62.5 µm 275 m (200)
220 m (160)
OSA-Express5S 1000BASE-Tc 0417 UTP Cat5 or 100 m IBM z15 T01b and
Cat6 IBM z14
OSA-Express4S 10GbE SRc 0407 MM 50 µm 550 m (500) IBM z15 T01b and
IBM z14b
MM 62.5 µm 275 m (200)
220 m (160)
Removal of support for OSA-Express 1000BASE-T featuresa: IBM z16 will be the last
IBM Z platform to support OSA-Express 1000BASE-T features. In the future, valid OSA
CHPID types will be supported only by OSA-Express GbE features, and potentially higher
bandwidth fiber Ethernet adapters.
a. All statements regarding IBM plans, directions, and intent are subject to change or withdrawal
without notice. Any reliance on these Statements of Direction (SoDs) is at the relying party's
sole risk and will not create liability or obligation for IBM.
Table 5-3 shows the CHPID types and number of OSA-Express features that are supported
on IBM z16 platforms.
OSA-Express7S 1 1 48 48 OSD
25GbE SR1.1cd
OSA-Express7S 1 1 48 48 OSD
25GbE SRcd
OSA-Express7S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SRc
OSA-Express6S 2 2 96 / 96 48 / 48 OSD
GbE LX/SXc
OSA-Express6S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SRc
Table 5-4 shows the CHPID types and number of OSA-Express features that are supported
on IBM z15 platforms.
OSA-Express7S 1 1 48 48 OSD
25GbE SR1.1b
OSA-Express7S 1 1 48 48 OSD
25GbE SRc
OSA-Express7S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SRc
OSA-Express6S 2 2 96 / 96 48 / 48 OSD
GbE LX/SXc
OSA-Express6S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SRc
OSA-Express5S 2 2 96 / 96 48 / 48 OSD
GbE LX/SXf
OSA-Express5S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SRg
Table 5-5 lists the CHPID types and number of OSA-Express features that are supported on
IBM z14 platforms.
Table 5-5 IBM z14 M0x and IBM z14 ZR1 supported OSA I/O features
I/O feature: Ports per Ports per Max. number ofa CHPID definition
z14 / z14 ZR1 feature CHPID
Ports I/O slots
OSA-Express7S 1 1 48 48 OSD
25GbE SR
OSA-Express6S 2 2 96 / 96 48 / 48 OSD
GbE LX/SX
OSA-Express6S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SR
OSA-Express5S 2 2 96 / 96 48 / 48 OSD
GbE LX/SX
OSA-Express5S 1 1 48 / 48 48 / 48 OSD
10GbE LR/SR
OSA-Express4S 1 1 NA / 48 NA / 48 OSD
10GbE SR/LR
OSA-Express7S
1000BASE-T
1000BASE-T
25 GbE
10GbE
10GbE
10GbE
GbE
GbE
GbE
Jumbo frame support x x x x x x x x x x x
(8992-byte frame size)a
QDIO Diagnostic x x x x x x x x x x x
Synchronization for z/OS
VIPA x x x x x x x x x x x
IPv6 x x xa x x xa x x x x xa
OSA-Express7S
1000BASE-T
1000BASE-T
25 GbE
10GbE
10GbE
10GbE
GbE
GbE
GbE
SNMP support for z/OS x x x x x x x x x x x
and Linux on IBM Za
ARP statisticsa x x x x x x x x x x x
ARP takeover x x x x x x x x x x x
IP network availability x x xa x x xa x x x x xa
Checksum offload x x xa x x xa x x x x xa
support for IPv4
Checksum offload x x xa x x xa x x x x xa
support for IPv6
QDIO OLM x x xa x x xa x x x x xa
Layer 2 support x x xa x x a
x x x x xa
EE x x x x x x x x x x x
IWQ x x xa x x xa x x x x xa
For example, with this support, possible bandwidth bottlenecks and root causes can be
analyzed.
5.4 Summary
The OSA features provide direct LAN connectivity as integrated features of the IBM Z
platform. They bring the strengths of IBM Z and z/Architecture to the modern network
environment.
Addresses
Non-QDIO (OSE)a,b
QDIO (OSD)
5.5 References
For more information about the OSA features and configuration, see the following
publications:
Communications Server: IP Configuration, SC27-3650
Communications Server: SNA Network Implementation Guide, SC27-3672
Open Systems Adapter-Express Customer’s Guide and Reference, SA22-7935
OSA-Express Implementation Guide, SG24-5948
Resource Measurement Facility Report Analysis, SC34-2665
The OSA-ICC function supports the TN3270 console (TN3270E), local non-System Network
Architecture (SNA) DFT 3270 emulation, 328x printer emulation, and 3215 console emulation
for TPF. This emulation support for console session connections is integrated in IBM Z
platforms.
IBM z16 A01 OSA-Express7S 1.2 GbE LX and SX (Feature Code 0454 and Feature Code
0455)
OSA-Express7S 1.2 1000BASE-T (Feature Code 0458)
OSA-Express7S GbE LXa and SXa (Feature Code 0442 and Feature Code
0443)
OSA-Express7S 1000BASE-Ta (Feature Code 0446)
OSA-Express6S 1000 BASE-Ta (Feature Code 0426)
IBM z16 A02 OSA-Express7S 1.2 GbE LX and SX (Feature Code 0454 and Feature Code
IBM z16 AGZ 0455)
OSA-Express7S 1.2 1000BASE-T (Feature Code 0458)
OSA-Express6S 1000 BASE-Ta (Feature Code 0426)
IBM z15 T01 OSA-Express7S GbE LX and SX (Feature Code 0442 and Feature Code 0443)
OSA-Express7S 1000BASE-T (Feature Code 0446)
OSA-Express6S 1000 BASE-Ta (Feature Code 0426)
OSA-Express5S 1000 BASE-Ta (Feature Code 0417)
Removal of support for OSA-Express 1000BASE-T hardware adaptersa: IBM z16 will
be the last IBM Z system to support OSA-Express 1000BASE-T hardware adapters
(Feature Code 0426, Feature Code 0446, and Feature Code 0458). The definition of all
valid Open Systems Adapter (OSA) channel path identifier (CHPID) types will be allowed
only on OSA-Express gigabit Ethernet (GbE) adapters, and potentially higher bandwidth
fiber Ethernet adapters, on future servers.
a. Statements by IBM regarding its plans, directions, and intent are subject to change or
withdrawal without notice at the sole discretion of IBM. Information regarding potential future
products is intended to outline general product direction and should not be relied on in making
a purchasing decision.
The OSA-ICC supports Ethernet-attached TN3270E consoles and provides a system console
function at IPL time and operating systems support for multiple logical partitions (LPARs).
Console support can be used by IBM z/OS, IBM z/VM, IBM z/VSE, and IBM z/Transaction
Processing Facility (z/TPF) software.
With the proper OSA-Express features, the OSA-ICC is configured on a port-by-port basis by
using the CHPID type OSC. When the CHPID shares two ports, the following server definition
rules apply:
Each physical port is defined to a unique IP network port number.
Different subnets are defined for each physical port host IP.
There is a single defined common gateway router with an interface on each IP subnet.
Only one of the IP network ports defines the IP default gateway.
Each CHPID can support up to 120 console session connections. These connections can be
shared among LPARs by using the multiple image facility (MIF) and can be spanned across
multiple channel subsystems (CSSs).
Removal of TLS 1.0 for OSA, Hardware Management Console (HMC), and Support
Element (SE): IBM z15 is the last IBM Z server to support the use of the Transport Layer
Security protocol 1.0 (TLS 1.0) for establishing secure connections to the SE, HMC, and
OSA-Integrated Console Controller (channel path type OSC).
Chapter 6. Console communications: IBM Open Systems Adapter-Express Integrated Console Controller 107
Figure 6-1 shows an example of the OSA-Express Integrated Console Controller in a single
system configuration.
LCSS 0 LCSS 1
MIF
OSC CHPID OSC CHPID
PCOMM PCOMM
Ethernet Ethernet
Subnet 22 Subnet 06
Console(s) Console(s)
Figure 6-1 Example of an OSA-ICC configuration
The following features are included in the base support for the OSA-ICC:
Local and remote session connectivity, depending on the provided security and
networking environment
Local and remote connections for configuration changes by using security features of the
IBM Z HMC environment
6.2 Connectivity
IBM Z platforms have base LIC support for the OSA-ICC function. At least one supported
OSA-Express feature must be installed.
The HMC or SE is used to create the configuration source file for the OSA-ICC CHPID and for
operation and diagnosis.
The OSA-Express7S 1.2-GbE LX and SX features (Feature Code 0454 and Feature Code
0466) support the OSC CHPID. These features are available on the IBM z16 platform.
The OSA-Express7S GbE LX and SX features (Feature Code 0442 and Feature Code 0443)
support the OSC CHPID. These features, which were introduced with IBM z15 T01, can be
carried forward to IBM z16 A01.
The OSA-Express6S 1000BASE-T (Feature Code 0417) Ethernet features support the OSC
CHPID. Feature Code 0417 is supported on the IBM z14 systems and as a carry forward
feature for IBM z15 and IBM z16.
Note: Certain functions might require specific levels of an operating system, program
temporary fixes (PTFs), or both. That information is provided when necessary within this
chapter. FIXCAT category information is available at IBM Fix Category Values and
Descriptions.
IBM Personal Communications 14 supports TN3270 Extensions that are used for OSA-ICC
support. Personal Communications is a component of the IBM Access Client Package for
Multiplatforms program suite. For more information, see this web page.
6.4 Summary
Here are the key characteristics of the Open Systems Adapter-Express Integrated Console
Controller (OSA-ICC) function:
A no-charge function that is integrated into the IBM z16, IBM z15, and IBM z14 LIC.
Each OSC CHPID can support up to 120 console session connections, can be shared
among LPARs, and can be spanned across multiple CSSs.
Support for TN3270 Enhancements (TN3270E) and local non-SNA DFT 3270 emulation
by using Ethernet-attached workstations.
Console support that can be used by z/OS, z/VM, z/VSE, and z/TPF.
Local non-SNA DFT 3270 and 328x printer emulation support that can be used by TSO/E,
CICS, IMS, or any other 3270 application that communicates through VTAM.
TLS/SSL with Certificate Authentication was added to the OSC CHPID to provide a secure
and validated method for connecting clients to the IBM Z host. Up to 48 secure sessions
per CHPID can be defined (the overall maximum of 120 connections is unchanged).
Chapter 6. Console communications: IBM Open Systems Adapter-Express Integrated Console Controller 109
The OSA-ICC function provides several potential benefits:
Simplicity
External controllers are no longer needed.
Scalable capacity:
– Facilitates the addition of partitions and operations support pools.
– Can be shared by up to 85 LPARs on IBM z16 A01, IBM z15 T01, and IBM z14 M0x.
– Can be shared by up to 40 partitions on IBM z16 A02, IBM z16 AGZ, IBM z15 T02, and
IBM z14 ZR1.
Improved availability:
– Can enable lights-out operation.
– Hot-plug OSA availability characteristics.
– OSA features are a highly integrated component of the IBM Z platform, with the
reliability, availability, and serviceability (RAS) characteristics inherent in IBM Z.
Low operating cost versus an external console controller:
– Power, cooling, cabling, and floor space requirements are minimized.
– No rack is necessary.
6.5 References
For more information about the OSA-ICC function, see the following publications:
Input/Output Configuration Program Users’s Guide for ICP IOCP, SB10-7177
Open Systems Adapter Integrated Console Controller User’s Guide, SC27-9003
OSA-Express Integrated Console Controller Implementation Guide, SG24-6364
Shared Memory Communications (SMC) on the IBM Z platform is a technology that can
improve throughput by accessing data faster with less latency while reducing CPU resource
consumption compared to traditional TCP/IP communications. Furthermore, applications do
not need to be modified to gain the performance benefits of SMC.
Important: Both SMC-R and SMC-D require an existing TCP link between the images that
are configured to use the SMC protocol. With the Shared Memory Communications
Version 2 (SMCv2), you can configure SMC over multiple subnets.
Both SMC protocols use shared memory-architectural concepts, which eliminate TCP/IP
processing in the data path while preserving TCP/IP quality of service (QoS) for connection
management purposes.
At the time of writing, Linux on IBM Z networking supports two Ethernet networking
connectivity options: the OSA-Express adapter family and the RoCE Express adapter
family. Using PCIe-based networking devices that are provided by the RoCE Express
adapter family aligns with the deployment model for Linux on other architectural platforms,
facilitates the usage of broader existing Linux ecosystem tools, and eases the effort to
enable exploitation of industry hardware optimizations and integrate into industry
software-defined networking models and tools, including Red Hat OpenShift Container
Platform.
Plan for your adoption of RoCE Express adapters for IBM Z networking connectivity. IBM
plans to continue to work toward common networking adapters for all operating systems on
IBM Z, IBM LinuxONE, and Linux on IBM Z.
a. All statements regarding IBM plans, directions, and intent are subject to change or withdrawal
without notice. Any reliance on these Statements of Direction (SoDs) is at the relying party's
sole risk and will not create liability or obligation for IBM.
There are two key requirements for RDMA, as shown in Figure 7-1:
A reliable lossless1 network fabric (LAN for Layer 2 in data center network distance)
An RDMA-capable NIC and Ethernet fabric
Host A Host B
1 Network switching infrastructure requirements have relaxed with the new RoCE Express3 features.
The ISM does not use queue pair (QP) technology like RDMA. Therefore, links and link
groups based on QPs (or other hardware constructs) are not applicable to ISM. The SMC-D
protocol has a design concept of a logical point-to-point connection that is called an SMC-D
link.
The ISM uses a virtual channel identifier (VCHID) similar to HiperSockets for addressing
purposes. Figure 7-2 shows the SMC-D LPAR-to-LPAR communications concept.
Figure 7-2 Connecting IBM z/OS LPARs in the same IBM Z platform by using ISMs
SMC-R is a hybrid solution (see Figure 7-3 on page 115) for the following reasons:
It uses a TCP connection to establish the SMC-R connection.
Switching from TCP to “out of band” SMC-R is controlled by a TCP option.
The SMC-R information is exchanged within the TCP data stream.
Socket application data is exchanged through RDMA (write operations).
Middleware/Application Middleware/Application
Sockets Sockets
TCP TCP
SMC-R SMC-R
IP IP
Interface Interface
IP Network (Ethernet)
Dynamic (in-line) negotiation for SMC-R is initiated by presence of TCP Option (SMCR)
TCP connection transitions to SMC-R allowing application data to be exchanged using RDMA
The hybrid model of SMC-R uses the following key existing attributes:
Follows a standard IP network connection setup.
Dynamically switches to RDMA.
The TCP connection remains active (idle) and is used to control the SMC-R connection.
Preserves critical operational and network management IP network features:
– Minimal (or zero) IP topology changes.
– Compatibility with TCP connection-level load balancers.
– Preserves the existing IP security model (for example, IP filters, policy, virtual local
area networks (VLANs), or Secure Sockets Layer (SSL)).
– Minimal network administrative or management changes.
Changing host application software is not required, so all host application workloads can
benefit immediately.
With IBM z15, the 25GbE RoCE Express2.1 and 10GbE RoCE Express2.1 features were
released. These features can be carried forward to the IBM z16.
The previous generation, 25GbE RoCE Express2 and 10GbE RoCE Express2, are available
with the IBM z14 and can be carried forward to IBM z16 and IBM z15.
The RoCE features are virtualized by the Resource Group LIC (firmware) that is running on a
processor that is internally characterized as an integrated firmware processor (IFP). There is
one IFP per system in IBM z15 and IBM z14, and two IFPs on IBM z16. The Resource Group
provides VFs for each RoCE Express adapter that is installed in the IBM Z server.
RoCE Express2 and later features support 126 VFs (63 per port). Older RoCE Express
10GbE supports only 31 VFs per feature and cannot be carried forward to IBM z16.
The 25GbE RoCE Express3 features are available in SR (Feature Code 0452) and LR
(Feature Code 0453) versions.
IBM z16 A01 supports a maximum of 16 RoCE Express features (in any combination), and
IBM z16 A02 and IBM z16 AGZ support a maximum of eight features. For 25GbE RoCE
Express3, the PCI Function IDs (FIDs) are associated with a specific (single) physical port
(port 0 or port 1).
For 25GbE RoCE Express2.1, the PCI FIDs are associated with a specific (single) physical
port (that is port 0 or port 1).
IBM z16 A01 supports a maximum of 16 RoCE Express features (in any combination), and
IBM z16 A02 and IBM z16 AGZ support a maximum of eight features. For 10GbE RoCE
Express3, the PCI FIDs are associated with a specific (single) physical port (port 0 or port 1).
Both ports are supported. For 10GbE RoCE Express2.1, the PCI FIDs are associated with a
specific (single) physical port (port 0 or port 1).
Both ports are supported. For 10GbE RoCE Express2, the PCI FIDs are associated with a
specific (single) physical port (port 0 or port 1).
A customer-supplied cable is required. The following types of cables can be used for
connecting the port to the selected 25 GbE switch or to another 25GbE RoCE Express
feature on the attached IBM Z platform:
25GbE RoCE Express3 and 2.x SR features:
– OM4 50-micron multimode (MM) fiber optic cable that is rated at 4700 MHz-km that is
terminated with an LC duplex connector (up to 100 meters, or 328 feet).
– OM3 50-micron MM fiber optic cable that is rated at 2000 MHz-km that is terminated
with an LC duplex connector (up to 70 meters, or 229.6 feet).
25GbE RoCE Express3 LR features: Single-mode (SM) 9-micron fiber optic cable for a
maximum connection distance of 10 km (6.2 miles).
A customer-supplied cable is required. The following cable types can be used for connecting
the port to the selected 10 GbE switch or to the RoCE Express feature on the attached IBM Z
platform:
10GbE RoCE Express3 and 2.x SR features:
– OM3 50-micron MM fiber optic cable that is rated at 2000 MHz-km. It terminates with
an LC duplex connector (up to 300 meters, or 984 feet).
– OM2 50-micron MM fiber optic cable that is rated at 500 MHz-km. It terminates with an
LC duplex connector (up to 82 meters, or 269 feet).
– OM1 62.5-micron MM fiber optic cable that is rated at 200 MHz-km. It terminates with
an LC duplex connector (up to 33 meters, or 108 feet).
10GbE RoCE Express3 LR features: SM 9-micron fiber optic cable for a maximum
connection distance of 10 km (6.2 miles).
The SMC-R and SMC-D are hybrid protocols that have been designed and implemented to
speed up TCP communication without changing applications.
SMC-R uses the RoCE Express features for providing (DMA) data transfer under the control
of an established TCP/IP connection (OSA). SMC-D can also use an established TCP/IP
connection over HiperSockets.
IBM z/OS 2.1 or later with program temporary fixes (PTFs) supports the SMC-R protocol with
RoCE. With IBM z/OS V2R4 and later, support for SMC-Rv2 is also provided.
Linux on IBM Z
For Linux on IBM Z, the Linux distribution partners included SMC-D and SMC-R support.
SMC running on Linux on IBM Z LPARs can be used also to communicate with LPARs
running z/OS. The following minimum requirements must be fulfilled:
RHEL 8
SUSE Linux Enterprise Server 12 SP4 (kernel 4.12.14-95.13.1 or higher)
SUSE Linux Enterprise Server 15 SP1
Ubuntu 18.10
Linux has also introduced support for SMC-Rv2 that is available in Linux kernel 5.10 or later,
and the following Linux distributions:
Ubuntu 21.04
RHEL 8.4
SUSE Linux Enterprise Server 15 SP3
For more information about RoCE features support, check with the distribution owners.
Linux on IBM Z
For Linux on IBM Z, the Linux distribution partners included SMC-D and SMC-R support.
SMC running on Linux on IBM Z LPARs can be used also to communicate with LPARs
running z/OS. The following minimum requirements must be fulfilled:
RHEL 8 or RHEL 9
SUSE Linux Enterprise Server 12 SP4 (kernel 4.12.14-95.13.1 or higher)
SUSE Linux Enterprise Server 15 SP1
Ubuntu 18.10, Ubuntu 20.04, or Ubuntu 22.04
Note: SMC (the existing architecture) cannot be used in the following circumstances:
Peer hosts are not within the same IP subnet and VLAN.
TCP traffic requires IPsec or the server uses FRCA.
The initial version of SMC (Shared Memory Communications Version 1 (SMCv1)) was limited
to TCP/IP connections over the same Layer 2 network. This version was not routable across
multiple IP subnets. The associated TCP/IP connection was limited to hosts within a single IP
subnet. The hosts required direct access to the same physical Layer 2 network (that is, the
same Ethernet LAN over a single VLAN ID). The scope of eligible TCP/IP connections for
SMC was limited to and defined by the single IP subnet.
SMCv2 supports SMC over multiple IP subnets for both SMC-D and SMC-R. It is also
referred to as SMC-Dv2 and SMC-Rv2. SMCv2 requires updates to the underlying network
technology. SMC-Dv2 requires ISMv2, and SMC-Rv2 requires RoCEv2.
The SMCv2 protocol is compatible with earlier versions so that SMCv2 hosts can continue to
communicate with SMCv1 hosts.
Although SMCv2 changes the SMC connection protocol to enable multiple IP subnet support,
SMCv2 does not change how the user TCP socket data is transferred, which preserves the
benefits of SMC to TCP workloads.
TCP/IP connections that require IPsec are not eligible for any form of SMC.
Chapter 8. HiperSockets
This chapter presents a high-level overview of the IBM HiperSockets capabilities as they
pertain to the IBM Z platform.
HiperSockets provides internal virtual local area networks (VLANs) that act like IP networks
within the IBM Z platform. Therefore, HiperSockets provides the fastest IP network
communication between consolidated Linux, IBM z/VM, IBM z/VSE, and IBM z/OS virtual
servers on a IBM Z platform.
The virtual servers form a VLAN. Using iQDIO, the communication between virtual servers is
through I/O queues that are set up in the system memory of the IBM Z platform. Traffic
between the virtual servers is passed at memory speeds. For more information about the
number of HiperSockets that are available for each IBM Z platform, see 8.2, “Connectivity” on
page 135.
This LIC function, which is coupled with supporting operating system device drivers,
establishes a higher level of network availability, security, simplicity, performance, and
cost-effectiveness than is available when connecting single servers or logical partitions
(LPARs) together by using an external IP network.
Note: Throughout this chapter, we provide operating system support information for the
functions that are described. Not every operating system supports all features. The
operating system support information is provided for IBM z/OS, IBM z/VM, IBM z/VSE,
IBM z/TPF, and Linux on IBM Z (supported distributions).
Regarding KVM hypervisor, KVM support is provided by Linux distribution partners. For
more information about KVM support for the IBM Z platform, see your distribution’s
documentation.
Consolidating that mid-tier workload onto multiple Linux virtual servers that run on an IBM Z
platform requires a reliable, high-speed network for those servers to communicate over.
HiperSockets provides that network. In addition, those consolidated servers also have direct
high-speed access to database and transaction servers that are running under z/OS on the
same IBM Z platform. This configuration is shown on the right side in Figure 8-1. Each
consolidated server can communicate with others on the IBM Z platform through
HiperSockets. In addition, the external network connection for all servers is concentrated over
a few high-speed OSA-Express adapters, and possibly RDMA over Converged Ethernet
(RoCE) interfaces.
Linux
Guest z/OS
z/OS Consolidation
Systems LPAR
z/VM LPAR
OSA-Express
TCP/IP network
TCP/IP network
Typically, before a packet can be transported on an external LAN, a LAN frame must be built.
Then, the Media Access Control (MAC) address of the destination host or router on that LAN
must be inserted into the frame. HiperSockets does not use LAN frames, destination hosts, or
routers. IP network stacks are addressed by inbound data queue addresses rather than MAC
addresses.
The IBM z16, IBM z15, and IBM z14 LICs maintain a lookup table of IP addresses for each
HiperSockets function. This table represents a VLAN. When an IP network stack starts a
HiperSockets device, the device is registered in the IP address lookup table with its IP
address and its input and output data queue pointers. If an IP network device is stopped, the
entry for this device is deleted from the IP address lookup table.
The controlling operating system that performs I/O processing is identical to OSA-Express in
QDIO mode. The data transfer time is similar to a cross-address space memory move, with
hardware latency close to zero. For total elapsed time for a data move, the operating system
I/O processing time must be added to the LIC data move time.
HiperSockets operations run on the processor where the I/O request is initiated by the
operating system. HiperSockets starts write operations. The completion of a data move is
indicated by the sending side to the receiving side with a Signal Adapter (SIGA) instruction.
Optionally, the receiving side can use dispatcher polling rather than handling SIGA interrupts.
The I/O processing is performed without using the system assist processor (SAP). This
implementation is also called thin interrupt.
The data transfer is handled much like a cross-address space memory move that uses the
memory bus, not the IBM Z I/O bus. Therefore, HiperSockets does not contend with other
system I/O activity in the system.
IBM Z Platform
LPAR LPAR LPAR
virtual virtual virtual
Server 1 Server 2 Server 3
Hardware assists
A complementary virtualization technology that includes the following features is available for
the IBM Z platform:
QDIO Enhanced Buffer-State Management (QEBSM)
Two hardware instructions that are designed to help eliminate the overhead of hypervisor
interception.
Host Page-Management Assist (HPMA)
An interface to the z/VM main storage management function that is designed to allow the
hardware to assign, lock, and unlock page frames without z/VM hypervisor assistance.
These hardware assists allow a cooperating guest operating system to start QDIO operations
directly to the applicable channel without interception by the z/VM operating system. This
process improves performance. Support is integrated into IBM Z. However, always check the
suitable FIXCATs.
Broadcast support
Broadcasts are supported across HiperSockets on Internet Protocol version 4 (IPv4) for
applications. Applications that use the broadcast function can propagate the broadcast
frames to all IP network applications that are using HiperSockets. This support is applicable
to Linux on IBM Z, z/OS, and z/VM environments.
The support of IPv6 on HiperSockets (CHPID type IQD) is available on the IBM Z platform,
and it is supported by z/OS and z/VM. IPv6 support is available on the OSA-Express7S,
OSA-Express6S, OSA-Express5S, and OSA-Express4 features in the z/OS, z/VM, and Linux
on IBM Z environments.
Support of guests is expected to be apparent to z/VM if the device is directly connected to the
guest (pass-through).
HiperSockets Network Concentrator support uses the next-hop IP address in the QDIO
header rather than a MAC address. Therefore, VLANs in a switched Ethernet fabric are not
supported. IP network stacks that use only HiperSockets to communicate with no external
network connection see no difference, so the HiperSockets support and networking
characteristics are unchanged.
To use HiperSockets Network Concentrator unicast and multicast support, a Linux distribution
is required. You also need s390-tools (for use with the IBM S/390® Linux kernel and device
drivers), which are available at this page.
Each HiperSockets device has its own Layer 2 MAC address and allows the use of
applications that depend on a Layer 2 address, such as DHCP servers and firewalls. LAN
administrators can configure and maintain the mainframe environment in the same fashion as
they do in other environments. This feature eases server consolidation and simplifies network
configuration.
A HiperSockets device can filter inbound datagrams by VLAN identification, the Ethernet
destination MAC address, or both. This feature reduces the amount of inbound traffic, which
leads to lower processor use by the operating system.
z/OS application workloads that are based on XML, HTTP, SOAP, Java, and traditional file
transfer can benefit from zIIP enablement by lowering general-purpose processor use.
When the workload is eligible, the HiperSockets device driver layer processing (write
command) is redirected to a zIIP, which unblocks the sending application.
With HS NTA, Linux on IBM Z can control the tracing of the internal VLAN. It captures records
in host memory and storage (file systems) that can be analyzed by system programmers and
network administrators by using Linux on IBM Z tools to format, edit, and process the trace
records.
With a customized HS NTA rule, you can authorize an LPAR to trace messages only from
LPARs that are eligible to be traced by the NTA on the selected IQD channel.
HS NTA rules can be set up on the Support Element (SE). There are four types of rules for the
HS NTA:
Tracing is unavailable for all IQD channels in the system (the default rule).
Tracing is unavailable for a specific IQD channel.
Tracing is allowed for a specific IQD channel. All LPARS can be set up for NTA, and all
LPARs are eligible to be traced by an active Network Traffic Analyzer.
Customized tracing is allowed for a specific IQD channel.
Note: IBM z/VM 6.2a or later, IP network, and Performance Toolkit APARs are required for
this support.
a. The earliest z/VM version in support at the time of writing is 7.1.
Incorporating the HiperSockets channel into the flat Layer 2 broadcast domain by using OSD
adapters simplifies networking configuration and maintenance. The virtual switch
HiperSockets bridge port eliminates the need to configure a separate next-hop router on the
HiperSockets channel to provide connectivity to destinations that are outside of a
HiperSockets channel. This configuration avoids the need to create routes for this internal
route in all hosted servers and the extra hop of a router to provide the Layer 3 routing
functions.
Figure 8-4 shows a sample Cross-CPC HiperSockets LAN configuration. This configuration
enables the creation of a single broadcast domain across CPCs and HiperSockets channels
at the same time by delivering a highly available configuration on both CPCs.
PR/S M
HiperSockets (IQD)
OSD OSD
CPC OP S2
In this configuration, VSwitch B in LPAR B and VSwitch D in LPAR D are the active bridge
ports that provide external connectivity between the external bridged IQD channel in CPC
OPS1 and the external IQD channel in CPC OPS2. This flat Layer 2 LAN essentially joins or
extends the HiperSockets LAN between CPCs across the external Ethernet network VSwitch
UPLINK port and VSwitch D UPLINK port.
Figure 8-5 shows a sample configuration of z/VSE and the Fast Path to Linux function. z/VSE
applications can directly communicate with a Linux IP network through the HiperSockets
without involving the IP network stack of z/VSE.
8.2 Connectivity
HiperSockets has no external components or external network. There is no internal or
external cabling. The HiperSockets data path does not go outside of one physical IBM Z
server.
HiperSockets is not allocated a CHPID until it is defined. It does not occupy an I/O drawer or a
Peripheral Component Interconnect Express (PCIe) I/O drawer slot. HiperSockets cannot be
enabled if all available CHPIDs on the IBM Z platform are used. Therefore, HiperSockets must
be included in the overall channel I/O planning.
Real LANs have a maximum frame size limit that is defined by their architecture. The
maximum frame size for Ethernet is 1492 bytes. Gigabit Ethernet (GbE) has a jumbo frame
option for a maximum frame size of 9 KB. The maximum frame size for HiperSocket is
assigned when the HiperSockets CHPID is defined. Frame sizes of 16 KB, 24 KB, 40 KB, and
64 KB can be selected. The default maximum frame size is 16 KB. The selection depends on
the characteristics of the data that is transported over a HiperSockets. The selection is also a
tradeoff between performance and storage allocation.
The MTU size that is used by the IP network stack for the HiperSockets interface is also
determined by the maximum frame size. Table 8-1 lists these values.
16 KB 8 KB
24 KB 16 KB
40 KB 32 KB
64 KB 56 KB
The maximum frame size is defined in the hardware configuration (Input/Output Configuration
Program (IOCP)) by using the CHPARM parameter of the CHPID statement.
z/OS allows the operation of multiple IP network stacks within a single image. The read/write
control I/O devices are required only once per image, and they are controlled by VTAM. Each
IP network stack within the same z/OS image requires one I/O device for data exchange.
Running one IP network stack per LPAR requires three I/O devices for z/OS (the same
requirement as for z/VM and Linux on IBM Z). Each additional IP network stack in a z/OS
LPAR requires only one more I/O device for data exchange. The I/O device addresses can be
shared between z/OS systems that are running in different LPARs. Therefore, the number of
I/O devices is not a limitation for z/OS.
An IP address is registered with its HiperSockets interface by the IP network stack when the
IP network device is started. IP addresses are removed from an IP address lookup table
when a HiperSockets device is stopped. Under operating system control, IP addresses can
be reassigned to other HiperSockets interfaces on the same HiperSockets LAN. This feature
allows flexible backup of IP network stacks.
Reassignment is possible only within the same HiperSockets LAN. HiperSockets is one
network or subnetwork. Reassignment is possible only for the same operating system type.
For example, an IP address that is originally assigned to a Linux IP network stack can be
reassigned only to another Linux IP network stack.
A z/OS dynamic virtual IP address (VIPA) can be reassigned only to another z/OS IP network
stack, and a z/VM IP network VIPA can be reassigned only to another z/VM IP network stack.
The LIC forces the reassignment. It is up to the operating system’s IP network stack to control
this change.
HiperSockets definition: The IBM z16 IOCP definitions for HiperSockets devices require
the keyword VCHID. Virtual channel identifier (VCHID) specifies the virtual channel identifi-
cation number that is associated with the channel path (type IQD). The valid range is 7C0 -
7FF.
Sharing of HiperSockets is possible with the extension to the multiple image facility (MIF).
HiperSockets channels can be configured to multiple channel subsystems (CSSs). They are
transparently shared by any or all configured LPARs without regard for the CSS to which the
partition is configured.
CSS 0 CSS 1
HiperSockets CHPID 04
8.3 Summary
HiperSockets is part of IBM z/Architecture technology and includes QDIO and advanced
adapter interrupt handling. The data transfer is handled much like a cross-address space
memory move, by using the memory bus. Therefore, HiperSockets does not contend with
other I/O activity in the system.
HiperSockets can be defined to separate traffic between specific virtual servers and LPARs
on one IBM Z server. Virtual private networks (VPNs) or network VLANs across HiperSockets
are supported to further isolate traffic as required. With integrated HiperSockets networking,
there are no server-to-server traffic flows outside the IBM Z platform. The only way to probe
these VLANs is by using the NTA function, and strict controls are required for that procedure.
The IBM Z platform supports up to 32 HiperSockets. Spanned channel support allows sharing
of HiperSockets across multiple CSSs and LPARs.
For more information about the HiperSockets virtual bridge support for z/VM, see z/VM
Connectivity, SC24-6267.
Coupling links support communication between z/OS and CFs. The CF provides critical
locking and serialization, data consistency, messaging, and queuing capabilities that allow the
systems in the sysplex to coordinate and share data.
A configured cluster has no single point of failure and can provide users with near-continuous
application availability during outages.
Figure 9-1 shows a possible Parallel Sysplex configuration with the Server Time Protocol
(STP) feature. STP provides time synchronization for multiple servers and CFs. The usage of
Network Time Protocol (NTP) servers as an External Time Source (ETS) is supported by
STP.
Figure 9-1 Sample Parallel Sysplex that uses stand-alone CFs and STP
IBM z16 supports improved time accuracy by connecting Precision Time Protocol (PTP)
(IEEE 1588) and NTP time networks directly to the central processor complex (CPC) drawers.
Pulse Per Second (PPS) support is available for the highest accuracy to the external time
reference for NTP time.
Note: STP is a mandatory hardware requirement for a Parallel Sysplex environment that
consists of more than one IBM Z machine.
STP is a facility that is implemented in the Licensed Internal Code (LIC) that presents a single
view of time to IBM Processor Resource/Systems Manager (IBM PR/SM).
A CF runs the Coupling Facility Control Code (CFCC) that is loaded into main storage at the
time the CF image is activated.
You can configure the CFs both in the processor where z/OS images run and in a dedicated
coupling-facility processor. The former is called an internal CF, and the latter is called a
stand-alone CF.
If you use an internal CF and the host CPC fails, both the z/OS images and the CF fail
simultaneously. In this situation, some structures might not be rebuilt in another CF until the
z/OS systems are recovered, which results in an application outage. Having a stand-alone CF
allows remaining systems to continue operating if a z/OS host fails. If the stand-alone CF
machine fails, the z/OS systems can quickly rebuild the structures on the backup CF to
minimize disruption to applications.
The alternative to using a stand-alone CF is to use duplex CFs across two machines. There is
an overhead to using this configuration, which is the tradeoff against having an extra machine
footprint for the stand-alone CF.
1
InfiniBand coupling and timing links are not allowed when an IBM z16 is a member in a Parallel Sysplex or CTN
configuration.
Note: Coupling Express LR (available on IBM z15 and older systems) is link-compatible
with Coupling Express2 LR (available on IBM z16).
If any messages will be transmitted across multiple sites through WDM products, ensure that
only qualified WDM products and supported configurations are used.
For a list of WDM vendor products that are qualified for coupling links (transporting CF or STP
messages) in a multi-site sysplex, see IBM Resource Link, which shows a list of qualified
WDM products (requires a Resource Link ID).
For availability purposes, use at least two coupling links to connect each server to each CF in
a Parallel Sysplex. Performance, availability, and distance requirements of a Parallel Sysplex
are the key factors determining the appropriate connectivity options for a particular
configuration.
Important: Parallel Sysplex supports connectivity between systems that differ by up to two
generations (N-2). For example, an IBM z16 can participate in an IBM Parallel Sysplex
cluster with other IBM z16, IBM z15, and IBM z14 systems.
Figure 9-2 Parallel Sysplex connectivity options (IBM z16, IBM z15, and IBM z14)
Table 9-1 lists the coupling link support for each IBM Z platform. Restrictions on the maximum
numbers can apply, depending on the configuration. Always check with your IBM support
team.
ICA SR1.1 Integrated 0176 8 GBps 150 meters 96* 48 96* 48* N/A N/A
Coupling (492 feet)
Adapter
ICA SR Integrated 0172 8 GBps 150 meters 96* 48 96* 48* 80* 16
Coupling (492 feet)
Adapter
HCA3-O InfiniBand 0170 10 kms N/A N/A N/A N/A 64* N/A
LRb Long Reach (6.2 miles)
(1 x
InfiniBand)
HCA3-Ob InfiniBand 0171 150 meters N/A N/A N/A N/A 32* N/A
(12 x (492 feet)
InfiniBand)
a. The maximum supported links depend on the IBM Z model or capacity feature code. These numbers are marked
with an asterisk (*).
b. InfiniBand coupling or timing links are not supported for connecting to IBM z16, IBM z15, or IBM z14 ZR1.
Notes:
The number of Peripheral Component Interconnect Express (PCIe) fanouts and ICA SR
ports that are available on a server depends on the number of CPC drawers, and in the
case of IBM z14 M/T 3907, the number of processing unit (PU) SCMs. Feature Code
0636 has one PU SCM and can place up to two PCIe I/O features in the CPC drawer.
Feature Code 0637 has two PU SCMs and can place up to four PCIe I/O features.
Feature Code 0638 and Feature Code 0639 have four PU SCMs and can place up to
eight PCIe I/O features.
On an IBM z16 A01, for example, there are 12 PCIe+ Gen3 fanouts per CPC drawer,
which means that there are a maximum of 24 ICA SR/ICA SR1.1 ports per drawer. So,
a 4-drawer IBM z16 A01 server can support up to 96 ICA SR/ICA SR1.1 ports.
An IC link is a fast coupling link that uses memory-to-memory data transfers. IC links do not
have physical channel ID (PCHID) numbers, but do require channel path identifiers (CHPIDs).
The ICA SR and SR1.1 are designed to drive distances up to 150 m and support a link data
rate of 8 GBps. They are designed to support up to four CHPIDs per port and seven
subchannels (devices) per CHPID.
For more information, see Planning for Fiber Optic Links, GA23-1409. This publication is
available at the Library section of Resource Link.
Note: Coupling Express2 Long Reach (Feature Code 0434) is link-compatible and can
connect to CE LR (Feature Code 0433).
Coupling Express LR uses 10-gigabit Ethernet (GbE) RDMA over Converged Ethernet
(RoCE) technology, can support distances up to 10 km unrepeated, and supports a link data
rate of 10 Gbps. For distance requirements greater than 10 km, clients must use WDM. The
WDM vendor must be qualified by IBM Z.
Coupling Express features support up to four CHPIDs per port and 32 buffers (that is,
32 subchannels) per CHPID. The Coupling Express feature is in the PCIe+ I/O drawer on
IBM z16, IBM z15, and IBM z14 ZR1, and in a PCIe I/O drawer in an IBM z14 M0x.
For more information, see Planning for Fiber Optic Links, GA23-1409. This publication is
available at the Library section of Resource Link.
2 PCIe+ I/O drawer (Feature Code 4023 on IBM z16, Feature Code 4021 on IBM z15, and Feature Code 4001 with
IBM z14 ZR1) is built in a 19” format with the PCIe cards oriented horizontally. All three features (Feature Code
4023, Feature Code 4021, and Feature Code 4001) can hold up to 16 I/O PCIe features. The PCIe I/O drawer on
an IBM z14 M0x cannot be carried forward to newer models.
Notes:
The IBM z14 M0x (M/T 3906) is the last IBM Z server to support InfiniBand coupling
connectivity.
InfiniBand coupling links are not supported on IBM z14 ZR1 (M/T 3907).
InfiniBand coupling links are high-speed links on IBM z14 M0x. The InfiniBand coupling links
originate from two types of fanouts:
HCA3-O (Feature Code 0171) for 12x InfiniBand links
HCA3-O LR (Feature Code 0170) for 1x InfiniBand links
Each fanout that is used for coupling links has an assigned adapter ID (AID) number that
must be used for definitions in the I/O configuration data set (IOCDS) to have a relationship
between the physical fanout location and the CHPID number.
Fanout adapter ID
Unlike channels that are installed in a PCIe I/O drawer, which are identified by a PCHID
number that is related to their physical location, InfiniBand coupling link fanouts and ports are
identified by an AID. The adapter ID (AID) value depends on its physical location. The AID
must be used to assign a CHPID to the fanout in the IOCDS definition. The CHPID
assignment is done by associating the CHPID to an AID and port. For more information, see
2.1.5, “Adapter ID” on page 22.
9.2.6 Dynamic I/O reconfiguration for stand-alone CF, Linux on Z and z/TPF
CPCs
With z14 Driver Level 36 (z14 GA2) and newer, support for dynamic activation of a new or
changed input/output definition file (IODF) on a stand-alone CF is supported without requiring
a power-on-reset (POR)/IML of the stand-alone CF CPC and without requiring the presence
of any z/OS or IBM z/VM image running an HCD instance on the same CPC.
3 Depending on the version of the HCD, the default setting for the subchannels is set to 7 or 32.
This support is applicable only when both the driving CPC and the target CPC are z16 with
the required firmware support (Bundle S24) and when the driving system’s z/OS is level 2.3 or
higher with APAR OA65559 installed.
The dynamic I/O for a stand-alone CF, Linux on Z or z/TPF CPC is shown in Figure 9.3.
HW
HCD*
Activation
Service
Figure 9-3 Stand-alone CF, Linux on Z and z/TPF HCD dynamic activation
Important: The Sysplex Timer menus on the Support Element (SE) were discontinued for
IBM z15 and later systems. The STP can be configured and managed from the HMC task
Manage System Time.
STP is a facility that is implemented in the LIC and presents a single view of time to Processor
Resource/Systems Manager (PR/SM) across multiple CPCs. Any IBM Z system can be
enabled for STP by installing the STP feature (Feature Code 1021). Each CPC that is planned
to be configured in a CTN must be STP-enabled.
It is possible to manually set the hardware clock when a stand-alone machine is powered on,
but for most customers it is not accurate enough. The systems in the CTN must obtain the
time from an ETS.
Tip: If you configured an STP CTN with three or more servers, see Important
Considerations for STP Server Role Assignments in IBM Docs.
The IBM z16 implements a card that combines the BMC and OSC (see Figure 9-4 on
page 151).
A Frame
CPC 1
To customer network*
(PTP or NTP connectivity)
CPC 0
Used PPS ports
Figure 9-4 z16 PTP, NTP, and PPS ports
On the IBM z16, the NTP client support is provided by a firmware function that is
implemented in the CPC. The code interfaces with the NTP servers. This interaction allows an
NTP server to become the single time source for the z16 CPC.
An ETS must be configured for the Current Time Server (CTS). Configuring the Preferred
ETS and Secondary ETS servers for the Preferred Time Server (PTS) and Backup Time
Server (BTS) reduces the risk of an STP-only timing network losing its time source.
An ETS does not need to be configured for systems that are not the PTS or BTS. If an ETS is
configured, its configuration is saved if the role of the system changes to the PTS or BTS.
For more information about how to configure ETS, see IBM Z Server Time Protocol Guide,
SG24-8480.
When PPS is used, a network (Ethernet LAN) connection to the ETS server is also required.
For more information about ETS support, see the Parallel Sysplex advantages web page.
Important: N-mode power signal can be enabled if both PTS and BTS are on the IBM
z16.
For more information, see IBM Z Server Time Protocol Guide, SG24-8480.
N-mode power sensing allows automatic failover of the CTS. To enable this feature, there
is a one-time setup step.
In the HMC “Manage System Time” task, change the automatic switchover function from
CPC1 to CPC2 for STP:
– After the function is enabled, the power subsystem (both Bulk Power Assembly and
intelligent power distribution unit (iPDU)) detects any power source loss (at the power
cord or power side level).
– If there is a failure, a signal is generated from the CPC1 to CPC2 for IBM z15 with
Integrated Battery Facility. The generation of this signal can take up to 30 seconds
depending on conditions.
Attention: If either of the two CTNs being joined and merged is restricted, the join
operation fails, which can lead to an outage. Therefore, for a successful join and merge
operation, none of the two CTNs shall be restricted.
A restricted or bounded CTN does not allow any CPC to be added or remove any CPC
from the CTN.
CTN split
When splitting a CTN, STP checks to ensure that the split does not result in a sysplex that
“splits” across the two resultant CTNs. New STP roles are automatically assigned in the
split-off CTN during the process. STP connectivity and max-stratum checking are performed
in both split-off CTNs. Checking results are previewed and confirmed before proceeding.
CTN Merge
When merging two CTNs, STP roles are automatically maintained in the merged-into CTN
while STP connectivity and max-stratum checking is performed in the merged CTN. Preview
and confirm the results before proceeding.
For more information about STP operating system support, see the STP tab on the Parallel
Sysplex web page.
9.4 References
For more information about understanding, planning, and implementing a Parallel Sysplex
cluster, see the Parallel Sysplex web page.
For more information about Parallel Sysplex, see the following publications:
Getting the Most Out of a Parallel Sysplex, SG24-2073
IBM Z Server Time Protocol Guide, SG24-8480
Planning for Fiber Optic Links, GA23-1409
Note: Not all features that are described in this chapter are supported on IBM Z platforms.
For more information regarding a specific feature, see the appropriate chapter.
In Table 10-1, a link is a physical connection over a transmission medium (fiber) that is used
between an optical transmitter and an optical receiver. The maximum allowable link loss, or
link budget, is the maximum amount of link attenuation (loss of light), expressed in
decibels (dB), that can occur without causing a possible failure condition (bit errors). When
you use multimode (MM) fiber, as the link data rate increases, the unrepeated distance and
link budget decrease.
The link budget is derived from combining the channel insertion loss budget with the
deallocated link margin budget. The link budget numbers are rounded to the nearest 10th of a
dB.
8 gigabits
per
N/A 10 km 6.4
second
Fibre Channel connection Single-mode (Gbps)
(FICON) Express LX (SM) 9 µm
16 Gbps N/A 10 km 6.4
500 50 m 1.68
500 35 m 1.63
FICON Express SX
MM 50 µm 16 Gbps 2000 100 m 1.86
500 35 m 1.57
500 82 m 1.8
10-GbE SR
MM 50 µm 10 Gbps 2000 300 m 2.6
2000 70 m 1.8
25-GbE SR MM 50 µm 25 Gbps
4700 100 m 1.9
Note: For more information about extended distances, see 10.4, “Wavelength-division
multiplexing” on page 163.
All feature types support a long wavelength (LX) laser version and an SX laser version. They
support native FICON (FICON channel-to-channel (FCTC)) and Fibre Channel Protocol
(FCP) channel modes.
For more information, see Table 10-1 on page 158, and Planning for Fiber Optic Links,
GA23-1409.
The repeated distance for a FICON channel is IBM Z qualified to a maximum of 100 km. For
all FICON features that use repeaters, the end-to-end distance between the FICON channel
and the CU is IBM Z qualified for up to 100 km (62 miles) only. RPQ 8P2981 is available for
customers who require distances over 100 km.
The links between the two directors are called Inter-Switch Links (ISLs). FICON Multihop
allows support for cascading up to four switches with three hops.
FICON Directors with ISLs add flexibility to the system configuration because they allow
one-to-many and many-to-many links to be defined.
Another way of extending the distance between a FICON channel and CU is with a Qualified
Wavelength Division Multiplexer (QWDM) infrastructure between the two sites (see 10.4,
“Wavelength-division multiplexing” on page 163).
FICON channel to CU end-to-end distance can be increased up to 100 km without a data rate
performance drop occurring if the FICON Director buffer credits are set. The number of buffer
credits that are required depends on the link data rate and the maximum number of buffer
credits that are supported by the FICON Director or CU, and application and workload
characteristics.
Although it is possible for FICON to maintain high bandwidth at distances greater than
100 km, these distances have not been qualified for use with IBM Z. They are achievable only
if enough buffer credits exist to support the link speed. Support for distances over 100 km can
be requested with RPQ 8P2981.
The maximum supported distance of the FICON channel path between two FICON Directors
is FICON Director vendor-specific. Each ISL requires one fiber trunk (two fibers) between the
FICON Directors.
Note: The transceiver type (LX or SX) at each end of a particular link must match.
WDM technologies
Other extended distance connectivity technologies are available to extend FICON and other
link types, for example, WDM. WDM technology also provides increased flexibility in that
multiple links and protocol types can be transported over a single dark fiber trunk. For more
information, see 10.4, “Wavelength-division multiplexing” on page 163.
1x InfiniBandb 10 km 5 Gbps
a. 150 meters distance is achieved by using OM4 fiber types only; with OM3 fiber, the distance is
100 meters maximum.
b. Cannot be configured if an IBM z16 is part of the Parallel Sysplex or Coordinated Time Network
configuration.
Combining Separating
Signals Different light frequencies Signals
over a single fiber path
Transmitters Receivers
Figure 10-2 WDM transmission technique
The actual signal bandwidth that the electronics can handle over one wavelength is a small
fraction of the inter-channel spacing. So, the signals do not interfere with one another and can
be multiplexed into a single fiber by using a passive grating multiplexer.
Historically, this solution was known as IBM Geographically Dispersed Parallel Sysplex.
Today, GDPS continues to be applied as a general term for a suite of business continuity
solutions, which include the ones that do not require a dispersed or multi-site sysplex
environment.
The GDPS solution is also independent of disk vendors if the vendor meets the specific levels
of IBM Metro Mirror, IBM Global Mirror, and IBM z/OS Global Mirror architectures. For more
information, see the GDPS web page.
IBM supports only WDM products that are IBM Z qualified for use in GDPS solutions. To
obtain this qualification, WDM vendors obtain licensed IBM patents, intellectual property, and
know-how that are related to the GDPS architecture. This access allows vendors to use
proprietary IBM protocols and applications that are used in a GDPS environment, which
include coupling and STP links, Metro Mirror, Global Mirror, and z/OS Global Mirror.
Licensing of IBM patents also provides the WDM vendor with technical information about
future IBM releases. Qualified vendors typically license this information for an extended
period, which allows them to subscribe to the latest GDPS architecture changes and be
among the first to market with offerings that support these features.
Note: Check with your WDM vendor for the current licensing status.
In addition, these vendor products are tested and qualified by IBM technicians with the same
test environment and procedures that are used to test the protocols that provide connectivity
for a GDPS configuration. This testing includes functions, recovery, and, in certain cases,
performance measurements.
Having access to these test facilities allows IBM to configure a fully functional sysplex and
simulate failure and recovery actions that cannot be tested as part of an operational customer
environment.
Components
The following GDPS components are used during the qualification process:
IBM Parallel Sysplex
IBM System Storage
Optical Wavelength Division Multiplexer (WDM)
IBM System Storage Metro Mirror (PPRC) (a synchronous form of remote copy)
IBM System Storage Global Mirror
IBM System Storage z/OS Global Mirror (XRC) (an asynchronous form of remote copy)
Protocols
The following GDPS connectivity protocols are tested during the qualification process:
FICON
FCP
FC ISLs
STP
Coupling Express LR links
1x InfiniBand coupling links
10 GbE and RDMA over Converged Enhanced Ethernet (RoCE and RoCE Express2)
using Shared Memory Communications - RDMA (SMC-R)
Often, these tested protocols are used in non GDPS environments too. The robust testing that
is performed during the qualification process provides a high level of confidence when you
use these IBM Z qualified optical WDM vendor products in non-GDPS environments.
Select Hardware products for servers in Resource Link, and then select the page that is
titled IBM System z® Qualified Wavelength Division Multiplexer (WDM) products for
GDPS solutions.
Note: It is important to select the particular WDM vendor link in Resource Link and
download the qualification letter to verify the details about the WDM product, model,
firmware level, and the IBM Z server models for which it is qualified.
10.5 References
For more information about fiber optic link distance, see the following publications:
Coupling Facility Channel I/O Interface Physical Layer, SA23-0395
Fiber Transport Services Direct Attach Planning, GA22-7234
Planning for Fiber Optic Links, GA23-1409
For more information about IBM Z connectivity, see this web page.
For more information about IBM Z qualified WDM vendor products, use this IBM Redbooks
publications search result.
The following cryptographic features and functions are part of the IBM Z environment:
Central Processor Assist for Cryptographic Function (CPACF). CPACF offers a set of
symmetric cryptographic functions for high encrypting and decrypting performance of
clear key operations. This interface is for Secure Sockets Layer/Transport Layer Security
(SSL/TLS), virtual private networks (VPNs), and data-storing applications that do not
require US Federal Information Processing Standard (FIPS2) 140-2 Level 4 security.
CPACF is implemented as a coprocessor in the processor core of the IBM Z platform.
The on-core design consists of one Compression Coprocessor (CMPCS), one CPACF,
and one IBM Integrated Accelerator for IBM Z Sort. The CPACF is embedded in each
processing unit (PU) core of the IBM Z PU chip.
The coprocessor supports SMT. For increased throughput, the IBM z14 coprocessor was
further developed so that the coprocessor results are now stored directly in the L1 cache
(on-core), which was carried over to later IBM Z processor generations.
Figure A-1 illustrates the on-core coprocessor. The highlighted (red rectangle) area shows
the functional blocks that belong to the CPACF. Leveraging this unit requires ordering the
CPACF feature (Feature Code 3863).
Crypto Express features are optional and available in different generations. All Crypto
Express features can be configured during installation as either a secure coprocessor or
as an accelerator. Changing the operational mode (coprocessor, Enterprise PKCS #11
(EP11), or accelerator) can be changed, but the process is disruptive to the feature
operation. This disruption can be avoided by using redundant features and planning and
configuration.
1
One of the industry-accepted PKCS that is provided by RSA Laboratories from RSA, which is the security division
of Dell EMC Corporation.
2 FIPS140-2 Security Requirements for Cryptographic Modules.
Note: When the Crypto Express8S PCIe adapter is configured as a secure IBM CCA
coprocessor, it still provides accelerator functions. However, you can achieve up to
three times better performance for those functions if the Crypto Express8S PCIe
adapter is configured as an accelerator.
Note: When the Crypto Express7S PCIe adapter is configured as a secure IBM CCA
coprocessor, it still provides accelerator functions. However, you can achieve up to
three times better performance for those functions if the Crypto Express7S PCIe
adapter is configured as an accelerator.
IBM z15 T01 and IBM z16 A01 support up to 30 Crypto Express7S (2 port) or up to 16
Crypto Express7S (1 port) features for up to 60 HSMs in supported combinations.
Note: Crypto Express7S and Crypto Express6S features can be carried forward to
IBM z16. Crypto Express5S is not supported on IBM z16.
Crypto Express6S
The Crypto Express6S feature is supported on IBM z16 and IBM z15 (carry forward only),
and IBM z14. It has the following characteristics:
It occupies one I/O slot in the PCIe I/O drawer and has one PCIe adapter (PCIeCC or
HSM), with one PCHID that is assigned to it according to its physical location.
Each Crypto Express6S PCIe adapter can be configured in one of the following modes:
– Secure IBM CCA coprocessor (CEX6C) for FIPS 140-2 Level 4 certification. This mode
includes secure key functions. The Crypto Express6s supports UDXs, which you can
use to define and load customized cryptographic functions.
– Secure IBM EP11 coprocessor (CEX6P) implements an industry-standardized set of
services that adhere to the PKCS #11 specification 2.20.
This cryptographic coprocessor mode introduced the PKCS #11 secure key function.
A TKE workstation is required to support the administration of the Crypto Express4S
when it is configured in EP11 mode.
– Accelerator (CEX6A) for acceleration of public key and private key cryptographic
operations that are used with SSL/TLS processing.
These modes can be configured by using the SE. The PCIe adapter must be configured
offline to change the mode.
Note: When the Crypto Express6S PCIe adapter is configured as a secure IBM CCA
coprocessor, it still provides accelerator functions. However, you can achieve up to
three times better performance for those functions if the Crypto Express6S PCIe
adapter is configured as an accelerator.
Up to 16 Crypto Express6S features are supported (16 PCIe adapters per supported
IBM Z platform).
Up to 85 domains on IBM z16 A01, IBM z15 T01, and IBM z14 M0x, and up to 40 domains
on IBM z16 A02, IBM z16 AGZ, IBM z15 T02, and IBM z14 ZR1 for LPARs or IBM z/VM
guests are supported. This enhancement is based on the new APXA, which enables the
z/Architecture to support up to 256 domains in an AP.
The Optica Prizm converter can be implemented in point-to-point, switched, and cascaded
FICON configurations, as shown in Figure B-1.
1. Point-to-Point FICON
FICON ESCON
ISL
FICON
ESCON
IP
FICON
Channel
Extension
IBM Z
without ESCON CHPID support
For more information about Prizm Protocol Convert, see the Optica website.
Note: IBM cannot confirm the accuracy of compatibility, performance, or any other claims
by vendors for products that have not been qualified for use with IBM Z platforms. Address
questions regarding these capabilities and device support to the suppliers of those
products.
IBM Facilities Cabling Services ESCON to FICON migration services can help you leverage
high-speed FICON to support an upgraded IBM Z environment. Also, it is possible to keep
using ESCON-attached devices to reduce migration costs.
The connector type for most fiber optic cable types is LC duplex, with the following
exceptions:
zHyperLink Express Multi-fiber Termination Push-on
(MTP) connector
12x InfiniBand Multifiber Push-On (MPO)
connector
Integrated Coupling Adapter Short Reach (ICA SR) MTP connector
1000BASE-T Ethernet RJ-45 connector (unshielded
twisted pair (UTP) copper
Ethernet cable)
For more information about fiber optic cables and connector types, see Appendix D, “Fiber
optic cables” on page 185.
The footnotes in Table C-1 reflect special considerations for certain features. Check the
referenced chapters for in-depth information.
The entry in this column always belongs to the supported platform information on the left.
For more information about support of extended distances, see Chapter 10, “Extended
distance solutions” on page 157.
zHyperLink Express1.1 0451 8 GBps MM 50 µm OM3 100 m (2000) IBM z16 and NB or carry
OM4 150 m (4700) IBM z15 forward
zHyperLink Express 0431 8 GBps MM 50 µm OM3 100 m (2000) IBM z16, Carry forward
OM4 150 m (4700) IBM z15, and
IBM z14
FICON Express16S SX 0419 16 Gbps MM 62.5 µm 15 m (200) IBM z15 and Carry forward
MM 50 µm OM2 35 m (500) IBM z14 or WDFM
OM3 100 m (2000)
OM4 125 m (4700)
FICON Express8S SX 0410 8 Gbps MM 62.5 µm 21 m (200) IBM z15, Carry forward
MM 50 µm OM2 50 m (500) IBM z14, and or WDFM
OM3 150 m (2000) IBM z14 ZR1
OM4 190 m (4700)
OSA-Express7S 1.2 1 GbE 0455 1 Gbps MM 62.5 µm 275 m (200) IBM z16 NB
SX
MM 50 µm OM3 550 m (500)
OSA-Express7S GbE SX 0443 1 Gbps MM 62.5 µm 275 m (200) IBM z15 T01 NB
IBM z16 A01 Carry forward
MM 50 µm OM3 550 m (500)
OSA-Express6S GbE SX 0423 1 Gbps MM 62.5 µm 275 m (200) IBM z15 T01 Carry forward
IBM z15 T02 NB or carry
MM 50 µm OM3 550 m (500) IBM z14 forward
IBM z16 NB
CF
OSA-Express6S 0426 100/1000 UTP Cat5 or 6 100 m IBM z15 T01 Carry forward
1000BASE-T Ethernet Mbps IBM z15 T02 NB or carry
IBM z14 forward
IBM z16 NB
Carry forward
RDMA over Converged Ethernet Chapter 7, “Shared Memory Communications” on page 111
(RoCE) and Shared Memory
Communications - Direct (SMC-D)
MM 50 µm OM2 82 m (500)
OM3 300 m (2000)
25GbE RoCE Express2 0430 25 Gbps MM 50 µm 70 m (2000) IBM z15 Carry forward
100 m (4700) IBM z16 Carry forward
IBM z14 NB
10GbE RoCE Express2 0412 10 Gbps MM 62.5 µm 33 m IBM z15 Carry forward
IBM z16 Carry forward
MM 50 µm OM2 82 m (500) IBM z14 NB
OM3 300 m (2000)
Coupling links Chapter 9, “Coupling links and common time” on page 141
Coupling Express Long 0433 10 Gbps SM 9 µm 10 km IBM z15 and NBc and
Reach (CE LR) IBM z14 Carry forward
ICA SR1.1 0176 8 GBps MM 50 µm OM3 100 m (2000) IBM z16 and NB and carry
OM4 150 m (4700) IBM z15 forward
ICA SR 0172 8 GBps MM 50 µm OM3 100 m (2000) IBM z16 Carry forward
OM4 150 m (4700) IBM z15 NB and carry
IBM z14 forward
NB and carry
forward
HCA3-O LR (1x InfiniBand) 0170 5 Gbps or SM 9 µm 10 km IBM z14 M0xd NBc , carry
2.5 Gbps forward, or
WDFM
HCA3-O (12x InfiniBand) 0171 6 GBps MM 50 µm OM3 150 m (2000) IBM z14 M0xd NBc , Carry
forward, or
WDFM
Crypto Express7S (2 port) 0898 N/A N/A N/A IBM z16 Carry forward
IBM z15 NB
Crypto Express7S (1 port) 0899 N/A N/A N/A IBM z16 Carry forward
IBM z15 NB
Crypto Express6S 0893 N/A N/A N/A IBM z16 Carry forward
IBM z15, Carry forward
IBM z14 NB
IBM zEDC Expresse 0420 N/A N/A N/A IBM z14 WDFM
a. Minimum fiber bandwidths in MHz/km for multimode (MM) fiber optic links are included in parentheses where
applicable.
b. Five km for a point-to-point link running at 32 GBps (direct connection to a switch or another device).
c. NB is a new build. Features can be added for an NB system or as carry forward during an MES or upgrade.
d. InfiniBand coupling and timing links cannot be used in a sysplex or Coordinated Timing Network (CTN) where
IBM z16 is a member.
e. The zEDC Express Peripheral Component Interconnect Express (PCIe) feature was replaced in IBM z15 and
newer IBM Z generations by the on-chip IBM Integrated Accelerator for zEDC.
Figure D-1 shows the following types of optical fiber that are used in a data center
environment with IBM Z platform:
Multimode (MM)
Single-mode (SM)
The difference between these modes is the way that light travels along the fiber. MM features
multiple light paths; SM features only one light path.
Cladding
125 µm dia.
DATA
LED = Core 50 or 62.5 µm
Light Emitting Diode dia
SX Laser =
Short Wavelength Laser
Outer coating
250 µm dia.
Single Mode (SM) Fiber
"Single path" for light to travel Cladding
125 µm dia.
DATA
LX Laser =
Core
Long Wavelength Laser
9 µm
diameter
Note: To keep the data flowing, thorough cleaning of fiber optic connectors is critical. Make
sure that you have the necessary fiber optic-cleaning procedures in place.
Figure D-2 shows the most common fiber cable connectors that are used in data center
environments.
E S C O N D u p le x
M P O c o n n e c to r
M T -R J L C D u p le x
Figure D-2 The connectors that are commonly used for optical cables
The MCP cable must be installed on both ends of a link and occupy the same space as a
standard 2-m jumper cable. Adapter kits containing the MCPs are available with either SC
Duplex connectors (to support coupling links) or ESCON connectors (to support ESCON to
Fibre Channel connection (FICON) migration). MCP adapters differ for 50 or 62.5 μm fiber.
MCPs reduce the maximum link distance to 550 meters for gigabit links.
Optical mode conditioners are supported for FICON, coupling links, and Open Systems
Adapter (OSA). For more information, see Planning for Fiber Optic Links, GA23-1409.
Important: One MCP cable must be plugged into the long wavelength transceiver at each
end of the link.
Fiber optic MCP cables cannot be ordered as product feature codes for IBM Z.
9 µm SM SC duplex connector
to to
50 µm MM ESCON duplex
receptacle
9 µm SM SC duplex connector
to to
50 µm MM SC duplex receptacle
9 µm SM SC duplex connector
to to
62.5 µm MM SC duplex receptacle
9 µm SM SC duplex connector
to to
62.5 µm MM ESCON duplex
receptacle
9 µm SM LC duplex connector
to to
62.5 µm MM SC duplex receptacle
9 µm SM LC duplex connector
to to
62.5 µm MM ESCON duplex
receptacle
Figure D-3 OM4 50/125 µm multimode fiber cable with MTP connectors
Custom cable lengths or standard cable lengths that are shown in Table D-2 are available
from IBM GTS or through other vendors, such as Anixter Inc.
00JA683 1 m (3.28’)
00JA684 2 m (6.56’)
00JA685 3 m (9.84’)
00JA686 5 m (16.40’)
00JA687 8 m (26.24’)
00LU282 10 m (32.80’)
00LU283 13 m (42.65’)
00JA688 15 m (49.21’)
00LU669 20 m (65.61’)
00LU284 40 m (131.23’)
00LU285 80 m (262.36’)
For more information, see Planning for Fiber Optic Links, GA23-1409. This publication is
available at the Library section of Resource Link.
9 µm SM LC Duplex Connector
to
SC Duplex Receptacle
50 µm MM LC Duplex Connector
to
SC Duplex Receptacle
9 µm SM SC Duplex Connector
to
LC Duplex Connector with
Coupler
Note: Fiber optic conversion kits are not orderable as product feature codes for IBM Z.
Fiber optic conversion kits can be ordered by using the IBM Networking Services fiber cabling
service options. Each conversion kit contains one cable.
References
For more information, see the following publications:
Coupling Facility Channel I/O Interface Physical Layer, SA23-0395
ESCON I/O Interface Physical Layer Document, SA23-0394
Fibre Channel Connection for S/390 I/O Interface Physical Layer, SA24-7172
Fiber Transport Services Direct Attach Planning, GA22-7234
Planning for Fiber Optic Links, GA23-1409
S/390 Fiber Optic Link (ESCON, FICON, Coupling Links and OSA) Maintenance
Information, SY27-2597
SBCON Single-Byte Command Code Sets zDAC IBM z/OS Discovery and Automatic
Connection Configuration
SCSI Small Computer System Interface zEDC IBM zEnterprise Data Compression
The publications that are listed in this section are considered suitable for a more detailed
description of the topics that are covered in this book.
IBM Redbooks
The following IBM Redbooks publications provide more information about the topics in this
document. Some publications that are referenced in this list might be available in softcopy
only.
Enterprise Extender Implementation Guide, SG24-7359
FICON Planning and Implementation Guide, SG24-6497
IBM Communication Controller for Linux on System z V1.2.1 Implementation Guide,
SG24-7223
IBM Communication Controller Migration Guide, SG24-6298
IBM HiperSockets Implementation Guide, SG24-6816
IBM z16 (3931) Technical Guide, SG24-8951
IBM z16 Technical Introduction, SG24-8950
IBM z14 (3906) Technical Guide, SG24-8451
IBM z14 Model ZR1 Technical Introduction, SG24-8550
IBM z14 Technical Introduction, SG24-8450
IBM z14 ZR1 Technical Guide, SG24-8651
IBM z15 (8561) Technical Guide, SG24-8851
IBM z15 Technical Introduction, SG24-8850
OSA-Express Implementation Guide, SG24-5948
You can search for, view, download, or order these documents and other Redbooks,
Redpapers, web docs, drafts, and additional materials, at the following website:
ibm.com/redbooks
Other publications
These publications are also relevant as further information sources:
Communications Server: IP Configuration, SC31-8513
Communications Server: SNA Network Implementation Guide, SC31-8777
Communications Server: SNA Resource Definition Reference, SC31-8565
Enterprise Systems Architecture/390 Principles of Operation, SA22-7201
Fiber Optic Link Planning, GA23-0367
Fiber Optic Links (ESCON, FICON, Coupling LInks and OSA) Maintenance Information,
SY27-2597
Online resources
These websites are also relevant as further information sources:
Fibre Channel Standard
– http://www.t10.org
– http://www.t11.org
FICON Director vendors
http://www.ibm.com/systems/storage/san/enterprise/index.html
IBM Parallel Sysplex
http://www.ibm.com/servers/eserver/zseries/pso
IBM Resource Link for documentation and tools
http://www.ibm.com/servers/resourcelink
IBM Z I/O connectivity
http://www.ibm.com/systems/z/hardware/connectivity/index.html
IBM Z networking
http://www.ibm.com/systems/z/hardware/networking/
SG24-5444-22
ISBN 0738461296
Printed in U.S.A.
®
ibm.com/redbooks