Building AI-Optimised Data Centre Networks: Interconnects, Latency and Throughput

Introduction

The gold rush of the 21st century is not for metal, but for floating-point operations per second. As organisations across India and the globe race to deploy large language models and generative tools, the underlying infrastructure is hitting a wall. Standard networking, designed for web traffic and occasional database queries, is proving insufficient for the relentless chatter of thousands of GPUs. The problem is that AI does not behave like traditional software. It moves in massive, bursty waves where a single delayed packet can stall a multi-million training job.

This blog explores the fundamental changes required to move from general-purpose facilities to high-performance hubs. It answers several critical questions:

• Why does traditional Ethernet struggle with AI?
• What are the three primary connectivity models for modern clusters?
• How do we manage the “Tail Latency” that threatens model accuracy?
• And finally, what does a professional checklist for AI readiness look like?

The Physics of AI Traffic

To understand data centre network architecture in the age of machine learning, one must first understand “Elephant Flows.” In a normal network, data travels in many small, manageable streams. AI workloads produce massive, long-lived flows that occupy the entire bandwidth of a link. If two of these flows collide on the same path, the network congests, and performance drops.

This is exacerbated by the “All-Reduce” operation. During training, GPUs must stop and share their findings with every other GPU in the cluster. The speed of the entire system is limited by the slowest link. If one switch port is congested, the other 511 GPUs in a cluster sit idle, wasting expensive compute time. This makes low latency and high throughput absolute requirements for AI data centre networks.

Three Models of AI Network Connectivity

Current industry standards have settled into three distinct approaches for connecting these high-density environments.

1. The InfiniBand Dedicated Fabric

InfiniBand remains a popular choice for massive-scale training. It is a lossless network, meaning it is designed at the hardware level never to drop a packet. By using credit-based flow control, it ensures that a sender only transmits data when the receiver has the buffer space to handle it. This creates a highly predictable environment, though it often requires specialised hardware and expertise.

2. RoCE (RDMA over Converged Ethernet)

Many enterprises prefer to stick with Ethernet because of its ubiquity. RoCE allows for Remote Direct Memory Access, which lets a GPU read data from the memory of another GPU without involving the CPU of either server. This bypasses the traditional software bottlenecks of the operating system, providing the speed of InfiniBand with the familiarity of Ethernet.

3. The Backend-Frontend Split

Modern architecture often separates the network into two. The “Frontend” handles management and storage using standard leaf-spine designs. The “Backend” is a dedicated, high-speed rail specifically for GPU-to-GPU communication. By separating the traffic, engineers ensure that a routine data backup on the frontend does not interfere with a critical training step on the backend.

The Checklist

Building a facility for AI is a mechanical and digital challenge. Based on industry standards, here is a professional checklist for assessing readiness:

• Power Density: Can the racks support 40kW to 100kW? Traditional racks often peak at 10kW. AI hardware requires significantly more juice.
• Cooling Modality: Is there a path to liquid cooling? Rear-door heat exchangers or direct-to-chip liquid cooling are becoming necessary as air cooling reaches its physical limits.
• Non-Blocking Fabric: The architecture must be non-blocking, meaning every device can communicate with every other device at full rated speed simultaneously.
• Buffer Management: Switches must have deep buffers to handle the bursty nature of AI data without discarding packets.

Best Practices for Throughput & Latency

To maintain a healthy data centre network architecture, operators are moving away from standard routing protocols towards more aggressive traffic management.

One such practice is Adaptive Routing. In a standard network, a packet takes a pre-determined path. In an AI-optimised network, the switch looks at the congestion of all available paths in real-time and sends the data through the least crowded route. This prevents the “hot-spotting” that kills performance in large clusters.

Another focus is reducing Tail Latency (the P99 latency). It is not enough for the average speed to be fast. If 99% of packets are fast but 1% are slow, the 1% dictates the speed of the AI training round. Using technologies like Priority Flow Control (PFC) helps manage these outliers by giving AI traffic a fast lane over background tasks.

Invenia’s Data Centre Solutions

Designing and maintaining these complex environments requires a partner who understands both the physical and logical layers of the facility. Invenia provides comprehensive data centre services that address these specific modern challenges. Our expertise covers the full lifecycle of infrastructure, from initial installation and structured cabling to ongoing maintenance and decommissioning.

For organisations looking to deploy AI data centre networks, we offer the technical ground support needed to ensure that high-density racks are installed correctly and that the complex cabling required for backend fabrics is managed with precision. By focusing on the physical integrity of the network, we ensure the digital layer performs at its theoretical maximum. Explore our data centre services or contact us for more details!

Conclusion

The pacing toward AI-optimised networking is a fundamental change in how we think about moving data. It requires moving from best-effort delivery to guaranteed, high-performance synchronisation. By selecting the right interconnect model, whether it be InfiniBand or RoCE, and adhering to strict infrastructure checklists, organisations can ensure their hardware spends its time computing rather than waiting. As the demand for intelligence grows, the strength of the network will determine who leads the race.

FAQs

1. What is RDMA and why is it used in AI?
   RDMA (Remote Direct Memory Access) is like a private express tunnel between servers. In a standard setup, data must stop at the CPU to be processed before moving on. RDMA allows one computer to reach directly into the memory of another, bypassing the CPU entirely. This is vital for AI because GPUs must share massive datasets instantly without the middleman slowing them down.|
   

2. What is the difference between a Leaf-Spine and a traditional Three-Tier network?
   A traditional Three-Tier network is like an old building with one central lift; everyone must go to the ground floor to reach another department, creating a bottleneck. A Leaf-Spine architecture is a modern campus where every room has a direct path to every other room. This ensures that no matter which two GPUs are talking, they are always the same short distance apart.
   

3. Why is liquid cooling mentioned in a networking blog?
   AI hardware generates immense heat that fans often cannot manage. When chips get too hot, they throttle or slow down to prevent damage. Even with the fastest cables, if the hardware slows down to cool itself, your network performance vanishes. Liquid cooling pulls heat away more effectively than air, ensuring the network stays at full speed.
   

4. What is a Non-Blocking network?
   Think of a ten-lane motorway that narrows to two lanes at a bridge; even fast cars will hit a jam. A non-blocking network is a motorway that stays ten lanes wide the entire way. It means the internal capacity of the switch is high enough to handle every single port running at maximum speed simultaneously, preventing the hardware from becoming a bottleneck.