Tom Kolniak, Sr. Director of Marketing, ATTO Technology
The choice between using Fibre Channel (FC) or other protocols is dictated by several factors, one of them being whether a dedicated SAN is required and/or desirable. Ethernet SANs are implemented in the context of a converged network or at the minimum sharing of switch ports for multiple use cases. Although, we see a rampant use of dedicated iSCSI fabrics also. FC networks are always utilized as dedicated SANs. For some use cases a dedicated SAN is not an option. In others, the mission critical nature of the application alone justifies a dedicated SAN.
The protocol and fabric selection criteria include:
- How critical is application reliability
and uptime? - How many application instances need to access the shared storage simultaneously?
- What are the storage performance needs of each application instance?
- How much latency is acceptable?
- Will jitter affect the performance of the application?
Other important criteria include packet delivery reliability, cost and adoption.
Comparing the Various Fabric and Protocol Choices
While comparing the various protocols in an absolute sense is highly dependent on the properties of the network components, some general statements can be made. Since FC networks are dedicated SANs, FC is generally better in reliability, performance and latency. Below is a brief overview of the principal, existent SAN technologies.
Fibre Channel
As a dedicated storage network, FC SANs provide significantly better performance and reliability in most “real world” use cases than Ethernet-based alternatives. FC SANs are highly scalable and are immune to network congestion, enabling FC SANs to be built with thousands of nodes without impacting throughput, latency or jitter. These factors simplify both the performance optimization and the addition of storage to FC SANs, tasks that are significantly more difficult to accomplish on converged Ethernet networks due to the need to constantly retune the network as workloads change and/or storage is added.
Ethernet/iSCSI
iSCSI has use cases where cost for shared storage is the driving factor, and where performance and data delivery reliability are not critical. Throughput and jitter can depend on various factors including complex network settings, type of NICs and switches and topologies. The simplest network topologies, and yes dedicated iSCSI fabrics, usually fare better. While the theoretical scale of Ethernet is unlimited, iSCSI over Ethernet networks can start to experience congestion and “noisy neighbor” issues well before they reach even half of their theoretical throughput. These issues can negatively impact the actual throughput by causing retries which reduce network capacity (even on 100GbE networks) and reliability for storage applications. Dedicated Ethernet SAN’s have better performance characteristics for storage connectivity.
NVMe over Fabric Protocols
One new set of emerging protocols for converged networks are the various flavors of NVMe over Fabrics(NVMe-oF). These protocols include NVMe over Fibre Channel (FC-NVMe, NVMe over RDMA over Converged Ethernet (NVMe-oRoCE), and NVMe over TCP (NVMe-oTCP) for Ethernet. FC-NVMe runs on standard FC networks and has all of the advantages inherent in a dedicated SAN, including reduced latency. The primary issue with all NVMe over Ethernet protocols is the relative immaturity of the larger storage ecosystem, and overall maturity of the protocol for shared storage. Technologists are diligently working on these protocols in various standards bodies, it’s just a matter of time.
Picking the Right SAN
The choice of a fabric and protocol for a SAN can have a significant effect on application performance and availability. NVMe-based Ethernet protocols offer performance that approaches FC with lower CapEx and OpEx costs, but the effect non-storage traffic can have on network congestion may impact performance and reliability of the SAN, thereby affecting application reliability and uptime. The availability and maturity of NVMe over Fabric storage arrays and appliances must be kept in mind as well.
For enterprise applications where the size of the SAN is in the hundreds or even thousands of ports, or where application instances require both high throughput and low latency, FC SANs continue to be the most reliable storage network and the lowest risk choice. The ability to scale and grow FC SANs without impacting performance is a key value prop. Because FC SANs are dedicated to storage traffic, they can consistently provide high throughput, low latency and extremely low jitter. FC-NVMe offers higher reliability compared to NVMe based Ethernet protocols as customers can deploy FC-NVMe using their already existing FC SAN’s that they have deployed in their data centers.
