AIDC Networking

_{Scale-Up and Scale-Out: The Dual Engines of AI Networking}

_{1. Definitions}

_{Scale-Up (Vertical Expansion)}

Scale-Up improves the performance of a single compute node by increasing GPU count, memory bandwidth, or chip-to-chip interconnect performance.

‍

Key Characteristics:

• Tightly coupled hardware
• Extremely high bandwidth
• Ultra-low latency (nanoseconds)

Typical Technologies:

• NVIDIA NVLink / NVSwitch
• AMD UALink (evolved from Infinity Fabric)
• Broadcom Scale-Up Ethernet (SUE)

_{Scale-Out (Horizontal Expansion)}

Scale-Out increases total compute capacity by adding more servers or racks connected via RDMA-capable networks.

‍

Key Characteristics:

• Cost-effective
• Elastic scalability
• Higher latency (microseconds to milliseconds)

Typical Deployments:

• RoCEv2 Ethernet clusters
• InfiniBand HDR / NDR pods
• Multi-server AI Pods (e.g. DGX SuperPOD)

_{2. Fundamental Differences in Application Scenarios}

Scale-Up networks use memory-semantic load–store communication, while Scale-Out relies on message-passing semantics. This fundamental difference makes the two architectures inherently non-interchangeable.

_{Technical Differences Between Scale-Up and Scale-Out}

_{Latency Composition: Static vs Dynamic}

‍

Static Latency

• Defined by SerDes, FEC, and chip design

Dynamic Latency

• Caused by congestion, queue buildup, and flow control

_Scale-Up

• Bypasses traditional L2/L3 network stacks
• Uses credit-based flow control and link-level retransmission
• Avoids DSP/CDR processing found in optical modules
• Typical latency: < 100 ns

_Scale-Out

• Requires congestion control (ECN / PFC / DCQCN / UEC)
• Multi-hop routing increases jitter and tail latency
• Typical latency: hundreds of ns to milliseconds

_{Interconnect Media: Copper vs Optics}

At scale: Scale-Up uses copper, Scale-Out uses optics.

‍

_{Case Study: DGX A100 vs DGX H100 (256-GPU Pod)}

• DGX A100 relies on InfiniBand → Scale-Out cluster
• DGX H100 adds a second-tier NVSwitch → cabinet-level Scale-Up system

Both provide 256 GPUs, but communication characteristics differ drastically.

_{Scale-Up at Its Peak: NVIDIA NVL72 SuperNode}

NVIDIA NVL72 integrates 36 Grace CPUs and 72 Blackwell GPUs into a liquid-cooled cabinet, forming the first true cabinet-scale SuperNode.

‍

_{Scale-Up Interconnect Inside NVL72}

‍

NVLink 5 + NVSwitch Characteristics:

• Per GPU: 1.8 TB/s bidirectional bandwidth
• Total (72 GPUs): 129.6 TB/s
• 5,184 differential-pair copper connections
• Full-mesh connectivity across the cabinet

Performance Highlights:

• 18× higher bandwidth than 800G RDMA
• Eliminates ~100 ns of optical DSP/CDR latency
• Unified giant memory system:
  • 13.5 TB HBM
  • 17 TB CPU LPDDR5X via NVLink-C2C

NVL72 behaves as a multi-server, cabinet-level unified compute node.

‍

_{Scale-Out Expansion Beyond the Cabinet}

‍

A SuperPOD of 8× NVL72 units (576 GPUs) is built by:

• Equipping each GPU tray with four 800G CX8 RDMA NICs
• Connecting cabinets via InfiniBand or RoCEv2

Two-Tier Network Architecture:

1. Scale-Up inside each cabinet
2. Scale-Out across cabinets within the data center

_{Scale-Across: Multi-Data-Center AI}

Scale-Across extends AI networking beyond a single physical site.

‍

_{Why Scale-Across Is Necessary}

‍

Single data centers face hard limits:

1. Power availability
2. Cooling capacity
3. Physical land constraints

Example:

• One NVL72 consumes ~120 kW
• A 1-GW AI factory requires ~8,300 cabinets

This is infeasible within a single facility.

‍

_{NVIDIA Spectrum-XGS}

‍

Key Components:

• ConnectX-8 and BlueField-3 SuperNICs
• 800G / 1.6T optical modules
• ZR / ZR+ coherent optics (80+ km DCI)
• Hollow-Core Fiber (HCF)

HCF Benefits:

• ~30% lower latency
• ~50% lower attenuation

Scale-Across introduces a new inter-DC communication layer on top of traditional L3 architectures.

‍

_{Enabling Optical Technologies}

OCS is projected to capture >50% of traditional switch market share over time.

_{Systematic Comparison}

‍