Fibre Channel
Fibre Channel (FC) is a high-speed storage networking protocol designed specifically for storage traffic. FC provides: dedicated storage network (no IP overhead), hardware-enforced lossless delivery, and sub-millisecond latency. FC is the gold standard for mission-critical storage in financial services, healthcare, and large enterprises.
FC speeds: 16 Gb/s (16GFC), 32 Gb/s (32GFC), 64 Gb/s (64GFC). FC requires dedicated infrastructure: FC HBAs (Host Bus Adapters), FC switches (Brocade, Cisco MDS), and FC-capable storage arrays. Higher cost than Ethernet-based alternatives.
iSCSI
iSCSI (Internet Small Computer Systems Interface) runs SCSI commands over TCP/IP networks, enabling block storage access over standard Ethernet. iSCSI provides good performance at lower cost than Fibre Channel — no dedicated FC infrastructure required.
iSCSI performance: 10GbE iSCSI delivers 1 GB/s throughput; 25GbE delivers 2.5 GB/s; 100GbE delivers 10 GB/s. iSCSI latency is higher than FC (200–500 μs vs. 50–100 μs) due to TCP/IP overhead. Jumbo frames (9000 MTU) improve iSCSI performance by reducing CPU overhead.
NVMe-oF
NVMe-oF (NVMe over Fabrics) extends the NVMe protocol over a network, providing near-local NVMe performance for shared storage. Three transports: NVMe/TCP (standard Ethernet, 100–200 μs latency), NVMe/RoCE (RDMA over Ethernet, 20–50 μs), NVMe/FC (Fibre Channel, 20–50 μs).
NVMe-oF is replacing iSCSI and FC for new deployments in performance-sensitive environments. All major storage vendors (Pure Storage, NetApp, Dell) support NVMe-oF. NVMe/TCP is the most accessible — no specialized hardware required.
NFS
NFS (Network File System) provides shared file access over IP networks. NFS is the standard for Linux/Unix shared storage: home directories, shared project storage, and AI training data. NFSv4.1 with pNFS (parallel NFS) enables parallel access to multiple storage servers for higher throughput.
NFS is not suitable for high-performance block workloads (databases, VMs) — use iSCSI, FC, or NVMe-oF for those. For AI training, parallel file systems (GPFS, Lustre) provide much higher throughput than NFS.
Protocol Comparison
| Protocol | Latency | Throughput | Cost | Best For |
|---|---|---|---|---|
| FC 32GFC | 50–100 μs | 3.2 GB/s/port | High | Mission-critical databases |
| iSCSI 25GbE | 200–500 μs | 2.5 GB/s/port | Medium | General enterprise storage |
| NVMe/TCP 100GbE | 100–200 μs | 10 GB/s/port | Medium | High-performance block |
| NVMe/RoCE 100GbE | 20–50 μs | 10 GB/s/port | Medium-High | Latency-sensitive applications |
| NFS 100GbE | 500–2000 μs | 10 GB/s/port | Low | Shared file access |
Storage Network Design
Storage networks should be isolated from compute networks for performance and security. Dedicated storage VLANs or physical networks prevent storage traffic from competing with application traffic. For FC, a dedicated FC fabric is standard. For iSCSI and NVMe/TCP, dedicated storage VLANs on high-speed Ethernet are recommended.
Storage network redundancy: dual-path storage (two independent paths from each server to storage) provides both redundancy and load balancing. MPIO (Multipath I/O) software manages dual-path storage on the host side.