INCITS committees have not been idled by the pandemic. Sure, we all miss the face-to-face discussions, however progress using virtual meetings continues to be realized and the Fibre Channel (FC) market remains strong as a transition to 32GFC and 64GFC continues and the standards for 128GFC come together. This 6th blog edition for How Fibre Channel Standards are Made series is focused on new developments with the Fibre Channel Protocol for SCSI (FCP) that has been progressing through the INCITS T11 and T10 committees this past year. I confess to being a lab rat who enjoys in line protocol analysis methods for problem triage. One of my go to triggers for FCP is the ABORT (ABTS). After reading this blog you should understand that I am in search of new FCP triggers and would welcome your recommendations.
While Peripheral Component Interconnect Express (PCIe) based Non-Volatile Memory Express (NMVe®) Solid State Drives (SSDs) out-perform Serial-Attached SCSI (SAS) SSDs in Enterprise All Flash Arrays (AFAs), SAS SSD in AFAs dominate the installed enterprise storage market. The 6/12/24 Gb/s SAS-2, SAS-3, and SAS-4 SSDs demonstrate greater architectural and physical flexibility and connect to more device types than NVMe SSDs. Also, SAS SSDs are a tried-and-true technology with an established ecosystem that will continue to play a significant role in enterprise AFAs. Not all mission critical applications need millions of disaggregated IOPs as do hyperscale applications. Mission critical applications depend on mature technologies and are deployed when standards have stabilized. The time is not yet right for PCIe 4.0 NVMe SSDs as PCIe Gen 3.0/3.1 continues to satisfy a broader ecosystem. Most of the overall storage market does not currently have the specialized use case requirements that are bottlenecked by PCIe Gen 3.0/3.1 to warrant the increase in cost and higher power consumption of PCIe 4.0 NVMe SSDs and risk of adoption of quickly evolving standards. In addition, there are continued application requirements for random-write performance which enhanced SAS SSDs realize through overprovisioning.
Which leads me to pose the following question for discussion in this blog entry:
Do standardized FCP error handling enhancements matter as the dominant FC SAN AFAs SSD interface transitions to NVMe SSDs from SAS SSDs?
Systems and storage vendors continue to use FC to meet the demands of enterprise storage networks and mission critical applications and continue to drive valuable enhancements into T11 standards for continued interoperability. Basically, FCP and FC-NVMe are both important. From this perspective, the T11 and T10 technical committees worked collaboratively to incorporate Sequence Level Error Recovery (SLER) into the proposed fifth version of the Fibre Channel Protocol for SCSI ( FCP-5 ) standard. The result is an important example of INCITs T11 and T10 committees working together to specify significant standards enhancements. The proposed FCP-5 standard has achieved the significant INCITS request for comment (RFC) letter ballot milestone in both the T10 and T11 committees. INCITS issued this RFC letter ballot to both committees because this was a joint project between T10 and T11. INCITS 556-2020 Fibre Channel – Non-Volatile Memory Express–2 standardized by the T11 committee utilizes transport level error handling and does not always rely on SCSI or NVMe protocol layer error recovery, thus improving a FCP and FC-NVMe SAN’s predictable behavior in the midst of errors.
Optionally implemented, confirmed Exchange completion may assist SCSI initiator and target devices. The Information Unit (IU) based protocol enables queued state dependent operations to progress and avoids a need for slower processing of ABORT related SENSE data. Confirmed completion with SLER provides a transport error recovery mechanism in FC’s link layer. SLER is architected to be optionally supported as it is phased into common use by initiators and targets. At the upper layer protocol (ULP/SCSI) level, error recovery is normally performed after a command timeout period of 30 seconds. SLER is normally performed every 2 seconds at the transport level, and avoids the ULP/SCSI timeout. As such, any error detection and recovery is transparent to the ULP/SCSI level. SLER implementations of the approximately 2second error detection and handling work significantly faster and transparently to ULP/SCSI layer. Maintaining predictable SAN performance is a core strength of FC and implementations of the proposed FCP-5 standard reinforce this strength.
Today’s FC adapters have the benefit of being able to run the SCSI command set concurrently with the NVMe command set in the same adapter. The initiator and target adapters support of both SCSI FCP and FC-NVMe concurrently on the same port and utilizing the proposed FCP-5 and FC-NVMe-2 error handling protocols can now detect and handle errors in dramatically less time and with lower overhead utilizing common recovery procedures at the FC transport layer. FC adapter or switch hardware changes are not needed to support the proposed FCP-5 standard’s Flush-based or REC-based Sequence level error detection and recovery and may be supported with upgraded software.
FCP-2/3/4 specified error detection and recovery was developed back when in-order delivery was from N_Port to N_Port and frames were delivered in-order. Exchanges were not delivered out-of-order. With the advent of Exchange-based routing, frames are delivered in-order within an Exchange and Exchanges may be delivered out-of-order. An interaction between two FC ports is termed an “Exchange”. Many protocols (including FCP and FC-NVMe) use an Exchange as a single command and response. Individual frames within the same Exchange are guaranteed to be delivered in-order. Individual Exchanges may take different routes through the fabric. This allows the fabric to make efficient use of multiple initiator to target paths. FC utilizes Exchanges and associated Sequences within an Exchange for larger data transfers. Enhanced error recovery keeps error detection and recovery handling at the transport level Exchange-based Sequence. The protocol implements a FLUSH request to poll each outstanding Exchange at 2s intervals and response Basic Link Services (BLSs) as a way to detect an error and take corrective actions.
ULP/SCSI error detection use of a 30 seconds or more timeout followed by command ABORT and then cleanup has become problematic as interconnect speeds have increased and storage latencies have declined to sub-milliseconds. ABORT (ABTS-LS) is an FC-FS-6 protocol that recovers FCP_Port resources associated with an Exchange that is being terminated, either because of a task management request or an error. ABTS-LS is my favorite analyzer trigger and measure of goodness when not detected. ABTS-LS may be transmitted whether or not the FCP-5 devices have agreed to Sequence level error recovery. By embracing previous FCP versions with the proposed FCP-5 standard’s optional enhancements, adoption of improved error handling can be phased into FC SANs. The proposed FCP-5 carries forward the ULP/SCSI abort and retransmission protocol and optionally, because of Exchange based routing, REC-based error detection and recovery which utilizes the Read Exchange Concise (REC) ELS (Extended Link Service) and Sequence Retransmission Request (SRR) FC-4 link service. FLUSH-based error detection and recovery operates within the Exchange and does not depend on ELSs or FC-4 link services for recovery operations. Error recovery capabilities are communicated and discovered and mutually accepted by initiators and targets during Process Login, so that legacy and enhanced error handling capabilities can be supported concurrently.
FC-NVMe-2 defined SLER protocol allows error recovery by using Sequence re-transmissions. This is accomplished via new Basic Link Services (FLUSH, FLUSH_RSP, and Responder Error Detected – RED) and FCP Device Control IUs (Sequence Retransmission request (FCP_SR) and Sequence Retransmission response (FCP_SR_RSP) Information Units (IUs) Exchanged between initiator and target ports to indicate, and recover from, any Sequence level errors during an Exchange. Intending to detect and recover from errors in a manner befitting a low-latency transport, FCP-5 and FC-NVMe-2 may not rely on the ULP SCSI or NVMe layer error recovery. These new and enhanced mechanisms include:
REC-based error detection and recovery
The Read Exchange Concise (REC) ELS may be used by the initiator FCP_Port to determine the state of an ongoing Exchange. Support for the REC ELS by both the initiator FCP_Port and target FCP_Port is indicated by the REC SUPPORT bit in the PRLI ELS request FCP Service Parameter page and the PRLI ELS accept FCP Service Parameter page. If the target FCP_Port responds with the REC SUPPORT bit set to one and an error is identified by any of the detection mechanisms defined in the FCP standard, then the initiator FCP_Port may use the REC ELS to determine the nature of the error. Target FCP_Ports that do not support the REC SUPPORT bit indicate they do not support the REC ELS by returning a Link Service Reject (LS_RJT) with a Reason Code of Command not supported, in response to an REC ELS. If an error is identified by any of the mechanisms defined in the FCP standard, then the initiator FCP_Port may request retransmission using the Sequence Retransmission Request (SRR) FCP_LS request.
FLUSH-based error detection and recovery
The FLUSH BLS and FLUSH_RSP BLS may be used by the initiator FCP_Port to determine the state of an ongoing Exchange. The FLUSH SUPPORT bits in the PRLI ELS request FCP Service Parameter page and PRLI ELS accept FCP Service Parameter page are used to determine support for FLUSH-based error detection and recovery. If an initiator FCP_Port and a target FCP_Port both indicate support of FLUSH BLSs, and both indicate support of confirmed completion, then FLUSH-based error detection and recovery is successfully negotiated. In this case, FLUSH-based error detection and recovery, confirmed completion, and task retry identification are supported. If an error is identified by any of the mechanisms defined in the FCP standard, then the initiator FCP_Port may request retransmission using the Sequence Retransmission request (FCP_SR) IU.
When the protocol confirms a lost frame, the initiator FCP_Port may request retransmission using the Sequence Retransmission request IU (FCP_SR IU). The FLUSH request is periodically transmitted by an initiator port to poll each outstanding Exchange to determine if a command is progressing properly and if any Sequences have been received incorrectly. Since multiple frames can be bundled into a Sequence, a Sequence can be used to transfer a large amount of data possibly up to multi-megabytes (instead of 2112 bytes for a single frame). The polling interval is controlled by a parameter known as the FLUSH Time-out-value (FLUSH_TOV) which has a specified default value ≥ 2 seconds. FLUSH response information from a target port is compared with the expected state information known by initiator port. If information is inconsistent, as per the FLUSH response IU, then error recovery actions may be performed to complete the Exchange.
The proposed FCP-5 standard has significant FCP and FC-NVMe SAN behavior benefits, compared to previous FCP versions. Use of confirmed completion using REC or FLUSH based error detection and recovery when deployed in server FC adapters and enterprise AFAs reinforces FC’s predictable performance tenant for mission critical applications. AFAs continue to utilize current and future SAS SSD technologies as a migration to NVMe SSDs continues. FC SANs with mixed FCP and FC-NVMe connected devices can employ FCP-5 and FC-NVMe-2 concurrently.
Back to the question I posed: Do standardized FCP error handling enhancements matter as the dominant FC SAN AFAs SSD interface transitions to NVMe SSDs from SAS SSDs? I think you will agree that the answer is yes. The ability to have a common enhanced error recovery and handling transport level along with the legacy error handling enables a seamless transition to FCP-5 and FC-NVMe-2. A fair assertion is that the proposed FCP-5 standard matters to the entire FC SAN ecosystem because errors do occur and speedy transparent recovery facilitates predictable behavior and performance for mission critical applications.
As I mentioned, this blog is part VI of our FCIA series on how Fibre Channel Standards are made. I encourage all readers to check out the other five blogs in the series here.
