The Fibre Channel Industry Association (FCIA) recently took on the topic of Fibre Channel (FC) Cabling in a live webcast. During the presentation, our experts Zach Nason, Data Center Systems; Greg McSorley, Amphenol Corporation and Mark Jones, Broadcom, answered frequently asked questions, some of which are presented in this blog. If you missed the live event, watch it on-demand to receive more in-depth information here.
This document is for anyone with questions concerning cabling in a FC environment, specifically those who are directly or indirectly responsible for SAN cable plant design based on the 16GFC and Gen 6 FC SAN platforms.
Q. How do I design a cabling infrastructure for a FC Storage Area Network?
A. Every cabling system is unique. This is due to variation in the architectural structure of the building which houses the cabling installation and a variety of other factors such and what type of data will be moving over the infrastructure. The design goal should be to ensure simplicity for easier management, future expansion and, serviceability.
Once everything is connected, FC begins to assign names to devices and practically runs automatically from a cabling perspective. You need to plan your FC SAN network topology by deciding what size is needed based on data stored, expected traffic across the network, and how much change will occur as far as adding or subtracting devices onto the network.
It’s important to keep in mind the links in between the devices may involve multiple cables and connections. A physical cabling topology can look very different than a logical network topology.
Finally, every data center’s cabling infrastructure is on a spectrum between a fully scalable, manageable and structured methodology and a completely congested, inconvenient and unstructured methodology. All cable plants are unique and variable to the network, and no cable plant is perfect. No matter which design you choose, the goal is to ensure its simplicity for easier management, future changes, and serviceability.
Q. What are the common cabling components of FC?
A. One of the nice things about FC is the simplicity and how we’ve designed it from generation to generation. At the box, two basic interconnects have been used for years: SFP and QSFP. You can use those in the intra-datacenter link over TwinAx™ cables or multi-mode fiber. For your inter-data center and mainframe links, you would use single mode fiber.
Q. What is structured cabling?
A. Structured connectivity in FC environments allows for rapid connection and cabling management of switches to servers and storage and enables data centers to plan for evolution and growth of IT infrastructure. Structured cabling consists of a number of standardized smaller elements (hence structured) called subsystems. Within those subsystems, intermediate link connections are made at patch cables to accommodate reconfiguration. The number and quality of these connections determine the connector loss, or loss of signal power of the link.
Q. Are there standards for network cabling?
A. Network cabling standards are used internationally and are published by ISO/IEC, CENELEC and the Telecommunications Industry Association (TIA). In the United States, the ANSI/TIA-568-D and ANSI/TIA-942B standards help to ensure a proper cabling installation.
Q. Why is structured cabling necessary?
A. The result of structured cabling is that all switches, servers and storage throughput throughout the data center are represented by individual port in front of patch cables in a centralized patch location, often called a main distribution area (MDA). With a correctly installed system, your requirements of today and tomorrow will be catered for and whatever hardware you choose to add will be supported.
Q. Are there copper cables in FC?
A. Yes, there are copper cables in FC. Copper cabling has been in FC since the beginning. We are not talking about Twisted pair or category cable. We’re talking about a Twin-Axial type of cable, which has much less insertion loss up to much higher frequencies. It’s limited as the speeds go up, but there are still ways to do it and, it’s an inexpensive way to complete a lot of internal rack connections. We’ll continue to use copper in the future.
Q. What is fiber and why do we use it?
A. Fiber is ultimately just a “waveguide for light” – light that goes in one end, comes out the other end. It’s most commonly made of glass/silica, but can also be plastic. We use fiber because it’s very low-cost to produce, extremely light and flexible, carries a tremendous amount of information and can easily carry large numbers of completely independent signals over the same fiber strand, without interference. Fiber can also carry signals thousands of kilometers without regeneration. Technology continues to radically improve what we can do with existing fiber infrastructure, without digging or disruption.
Q. What do we actually send over fiber?
A. Our digital signals must be encoded into analog pulses of light. The simplest and cheapest method is known as “IMDD” or Intensity Modulation with Direct Detection. They are typically encoded as “NRZ” or Non-Return to Zero, which is really just a fancy way of saying “bright for a 1, dim for a 0.” This modulation (called “baud”) can happen billions of times per second. Beyond 25GBaud, this technique gets increasingly hard to scale. “Better than NRZ” systems are becoming more pervasive. PAM4 (Pulse Amplitude Modulation) QSFP28 optics are starting to ship today.
Q. Can I use the same connectors when the next generation or speed of FC increases?
A. Yes! The same form factor has been used since 1GFC. At every speed increase, there have been tweaks to the certain area of the connector and plug, but the same form factor. The SFP and the QSFP are the basic building blocks for most boxes. You may lose some distances on some of your copper interconnect in the rack, but it’s usually minimal from speed-to-speed.
Q. What is the difference between multi-mode and single mode fiber types?
A. Multi-mode fiber (MMF) is specifically designed for use with “cheaper” light sources because it allows you to use less precise transceivers. It has an extremely wide core, allowing the use of less precisely focused, aimed, and calibrated light sources. But it comes at the expense of long-distance reach. OMI/OM2 MMF is found with orange fiber jackets and was originally designed for 100M/1310nm signals, starts to fail at 16G speeds. OM3/OM4 MMF (aka “laser optimized”) is found with aqua fiber jackets and was specifically designed for modern 850nm short reach laser sources and supports 16G signals at much longer distances. It is required for 32G/128G signals (which are really both 32G signals), 100m.
Single mode Fiber (SMF) is used for high bandwidths and long distances. SMF has a much small core size, between 8-10 microns. It has no inherent distance limitations caused by modal dispersion. SMF has an even broader array of types than MMF. OS1 “tight buffered” is for indoor use, OS2 “loose” is for buried use. Classic SMF can be called “SMF-28”, (a Corning product name), but it also comes in many different formulations such as Low Water Peak Fiber (LWFP), Dispersion Shifted Fiber (DSF), Non-Zero Dispersion Shifted Fiber, etc. etc.
Q. Why exactly is bit error rate important in FC?
A. Because it measures the data transmission performance of your systems. You want your transmission to be as lossless as possible, and one decimal out of place can cause a lot of trouble.
There are different definitions. Bit Error Rate (BER) refers to the number of errors during a certain time frame. Bite Error Ratio is the FC definition, which means the number of bits output from a receiver that differs from the correct transmitted bits, divided by the number of transmitted bits. A bit error is any received bit that has been distorted in some way.
FC requires a raw BER of less than or equal to 10-12 up to 16GFC. 32GFC calls for a raw BER of less than or equal to 10-6 because at 32GFC, 256B/257B encoding which includes Forward Error Correction (FEC). Combined, this gives a BER of better than 10-12. When using 10GBASE-T, FCoE requires Cat6a or Cat7 to meet 10-12 requirements.
For additional information on FC, please visit the Fibre Channel Industry website at https://fibrechannel.org/. This site contains resources such as QuickLinks to the FC roadmap, a library of all FC webcasts, our latest FC Solutions Guide and more.
