NVIDIA Mellanox MCX623106AN-CDAT Server Adapter in Action
June 10, 2026
A mid-sized cloud provider recently faced a familiar challenge: their legacy 25GbE NICs could not keep up with the explosive growth of NVMe-oF storage and distributed AI training workloads. CPU utilization often exceeded 70% on storage nodes due to traditional TCP/IP stack overhead, while tail latency during peak hours hurt application performance. This real-world case shows how the NVIDIA Mellanox MCX623106AN-CDAT transformed their infrastructure through RDMA and RoCE.
The provider's storage cluster consisted of 50 Ceph nodes, each handling both client I/O and internal replication traffic. With 3,000+ NVMe drives in the backend, three problems became acute: high CPU interrupts from network processing, inconsistent latency under mixed workloads, and east-west bandwidth saturation. The legacy adapters lacked hardware offloads for RoCE, forcing every packet to go through the kernel stack. After evaluating MCX623106AN-CDAT specifications and the MCX623106AN-CDAT datasheet, the team selected the MCX623106AN-CDAT ConnectX adapter PCIe network card for a pilot deployment.
The new architecture replaced legacy adapters with the NVIDIA Mellanox MCX623106AN-CDAT in eight storage nodes and four compute nodes. The MCX623106AN-CDAT Ethernet adapter card features dual-port 100Gb/s, PCIe 4.0/5.0, and native RoCE support. Key deployment steps included:
- Enabling DCQCN (Data Center Quantized Congestion Notification) for RoCE congestion control
- Partitioning PFC (Priority Flow Control) domains to isolate storage and AI training traffic
- Upgrading the fabric to a lossless Ethernet spine-leaf topology (25ms buffer threshold)
- Deploying NVMe-oF with RDMA for block storage access
The adapter's ASAP² offload engine processed over 90 million packets per second on the storage nodes, completely bypassing the host CPU for data movement. The MCX623106AN-CDAT Ethernet adapter card solution also enabled in-line encryption without performance penalty — a requirement for the provider's financial services tenants.
After three months of production use, the provider documented significant improvements:
| Metric | Legacy 25GbE (TCP) | MCX623106AN-CDAT (RoCE) | Improvement |
|---|---|---|---|
| Avg. latency (4KB random read) | 42 µs | 8 µs | -81% |
| CPU utilization (storage node) | 71% | 19% | -73% |
| Max throughput per node | 21 Gb/s | 98 Gb/s | +367% |
| Tail latency (99.9th percentile) | 218 µs | 34 µs | -84% |
The MCX623106AN-CDAT Ethernet adapter card also proved fully MCX623106AN-CDAT compatible with the provider's existing Cumulus Linux switches and Ceph BlueStore backend. No driver or firmware conflicts emerged during the staged rollout.
This deployment confirms that the NVIDIA Mellanox MCX623106AN-CDAT is not just a bandwidth upgrade — it fundamentally changes server efficiency through RDMA/RoCE. By offloading network processing and enabling lossless Ethernet, the adapter reduced storage latency by 5x while freeing CPU cores for application workloads. For IT managers evaluating MCX623106AN-CDAT price and ROI, the provider achieved hardware payback within seven months through reduced server count and lower licensing fees.
As more organizations adopt NVMe-oF, GPU direct RDMA, and microservices, the MCX623106AN-CDAT ConnectX adapter PCIe network card offers a proven path to low-latency, high-throughput data centers. Those seeking MCX623106AN-CDAT for sale can contact authorized NVIDIA Mellanox distributors. Full MCX623106AN-CDAT specifications and deployment guides are available in the official MCX623106AN-CDAT datasheet.