NVMe Fundamentals
NVMe (Non-Volatile Memory Express) is a storage protocol designed specifically for flash memory. Unlike SATA and SAS, which were designed for spinning disk, NVMe takes advantage of flash memory's low latency and parallelism. NVMe connects storage devices directly to the CPU via PCIe, eliminating the storage controller bottleneck.
Key NVMe advantages: queue depth of 65,535 (vs. 32 for SATA, 254 for SAS), multiple queues per CPU core, lower CPU overhead, and sub-100 microsecond latency for local NVMe SSDs.
NVMe vs. SATA/SAS
| Metric | SATA SSD | SAS SSD | NVMe SSD |
|---|---|---|---|
| Sequential read | 550 MB/s | 2,100 MB/s | 7,000+ MB/s |
| Random read IOPS | 100,000 | 200,000 | 1,000,000+ |
| Latency | 100–200 μs | 50–100 μs | 20–100 μs |
| Queue depth | 32 | 254 | 65,535 |
NVMe over Fabrics (NVMe-oF)
NVMe-oF extends the NVMe protocol over a network fabric, enabling shared NVMe storage with near-local performance. NVMe-oF maintains the low latency and high throughput of local NVMe while providing the shared access and management benefits of networked storage.
NVMe-oF transports: NVMe/RDMA (InfiniBand or RoCEv2 — lowest latency), NVMe/FC (Fibre Channel — enterprise SAN environments), NVMe/TCP (standard Ethernet — most accessible).
NVMe/TCP
NVMe/TCP runs NVMe-oF over standard TCP/IP networks, making it accessible without specialized RDMA hardware. It provides significantly lower latency than iSCSI (which also runs over TCP) because it eliminates the SCSI translation layer.
NVMe/TCP latency: 100–200 microseconds over 25GbE, compared to 200–500 microseconds for iSCSI. Not as low as NVMe/RDMA (20–50 microseconds) but accessible with standard Ethernet infrastructure. Supported by all major storage vendors (Pure Storage, NetApp, Dell).
NVMe/RDMA
NVMe/RDMA uses RDMA (Remote Direct Memory Access) for the lowest-latency NVMe-oF transport. Two options: NVMe/RoCE (RoCEv2 over Ethernet) and NVMe/IB (InfiniBand). Both provide 20–50 microsecond latency — approaching local NVMe performance.
NVMe/RDMA is used in high-performance environments where storage latency is critical: financial trading, real-time analytics, and AI training checkpoint storage. Requires RDMA-capable NICs and proper PFC/ECN configuration for RoCEv2.
Use Cases
- Database storage: NVMe all-flash arrays provide the IOPS and latency required for Oracle, SQL Server, and PostgreSQL at scale
- AI checkpoints: Local NVMe SSDs provide the fastest checkpoint write performance
- VDI boot storage: NVMe handles boot storms that overwhelm SATA/SAS storage
- Real-time analytics: NVMe enables in-memory-like performance for analytics workloads
- High-frequency trading: NVMe/RDMA provides the sub-100 microsecond latency required for trading systems
Deployment Considerations
- NVMe drives generate more heat than SATA/SAS — ensure adequate cooling in high-density deployments
- NVMe-oF requires host-side drivers — verify OS and hypervisor support before deployment
- NVMe/TCP is the easiest NVMe-oF deployment — no specialized hardware required
- NVMe/RDMA requires RDMA-capable NICs and proper network configuration
- Monitor NVMe drive health — NVMe SSDs have write endurance limits that must be managed