Glossary of Terms

Autoscaling automatically adjusts computing resources based on current workload demand.

A 16-bit floating-point format optimized for AI training with improved numerical stability versus FP16.

A collection of warps that execute together on a single streaming multiprocessor.

Saving model and optimizer state during training so work can resume after interruption.

Compute-bound vs memory-bound describes whether performance is limited by GPU arithmetic capacity or by how fast data can be fed to the GPU.

A container is a lightweight and portable software package that runs consistently across different machines.

NVIDIA’s proprietary programming platform that allows software to run computations directly on NVIDIA GPUs.

Data parallelism splits training data across multiple GPUs so each GPU trains on a portion of the dataset.

NVIDIA Data Center GPU Manager, used for GPU telemetry, health checks, and monitoring.

High-level AI software such as PyTorch, TensorFlow, or JAX that abstracts GPU programming.

Distributed training uses multiple GPUs or servers working together to train a single model faster or train models too large for one machine.

A 16-bit floating-point numeric format that improves performance and reduces memory and power usage.

A 32-bit floating-point numeric format used for higher-precision computation.

Low-level software that allows the operating system and applications to communicate with GPU hardware.

The layered memory system (registers, shared memory, caches, HBM, and host memory) that determines access speed and bandwidth.

Optimizing and compiling compute graphs for faster, more efficient execution on target hardware.

High Bandwidth Memory is stacked, ultra-fast memory located next to the GPU die to support large-model training and high data throughput.

Optimizing model weights using data and backpropagation to improve performance.

An 8-bit integer format commonly used to accelerate and compress AI inference.

A function that runs in parallel across many GPU threads.

Kubernetes is an orchestration platform that automates deployment, scaling, and management of containerized AI workloads.

A Large Language Model is an AI model trained on massive datasets to understand and generate human-like text.

A metric estimating how much of a GPU’s theoretical compute is actually achieved by a workload.

NVIDIA hardware feature that partitions one GPU into multiple isolated GPU instances.

Using multiple numeric precisions together to improve throughput and efficiency while preserving model quality.

A standard for message passing between processes in distributed computing.

Operating a cluster so multiple users or organizations can securely share the same infrastructure.

NVIDIA’s library for fast collective communication (e.g., all-reduce) across GPUs and nodes.

NVIDIA’s high-speed interconnect for GPU-to-GPU communication inside a server.

A measure of how fully an SM is utilized compared to its maximum capacity.

Software that manages job lifecycle, placement, scaling, failures, and recovery across a cluster.

A Pod is the smallest deployable unit in Kubernetes, typically containing one or more tightly coupled containers.

AMD’s open software platform for programming and running workloads on AMD GPUs.

The execution layer that manages GPU kernels, memory allocation, and scheduling during program execution.

Software that decides which workloads get which GPUs and when.

The core compute unit inside an NVIDIA GPU that executes threads and warps.

Specialized GPU hardware designed to accelerate matrix operations used in AI workloads.

Low-level software that allows the operating system and applications to communicate with GPU hardware.

The smallest unit of execution on a GPU.

Running a trained model to produce outputs (predictions, generations, classifications).

The Transformer is a neural network architecture built around attention mechanisms and used in most state-of-the-art language and vision models.

A memory model that lets CPU and GPU share a single memory address space.

A virtualized GPU that allows multiple users or workloads to share a single physical GPU.

A group of 32 threads on NVIDIA GPUs that execute instructions simultaneously.

Specialized hardware (GPU, TPU, or ASIC) that speeds up compute compared to CPUs.

A collective operation that aggregates values across workers and distributes the result to all workers.

Running workloads directly on physical servers without virtualization.

Resuming a job after failure by loading a previously saved checkpoint.

A scalable multi-stage network design widely used in modern data centers.

Traffic moving between servers inside the data center.

The ability to scale resources up or down based on demand.

The high-speed network connecting servers and enabling distributed training or inference.

The scope within which a single failure can impact service.

A high-bandwidth network topology designed to reduce congestion and provide near non-blocking performance.

Scheduling all parts of a distributed job simultaneously so it can start and run efficiently.

A server specifically designed to host and power multiple GPUs for AI and HPC workloads.

A Graphics Processing Unit is a chip built for massive parallel math operations, making it essential for training and running modern AI models.

A cluster containing multiple types or generations of GPUs or accelerators.

A cluster built from a single consistent server and GPU configuration.

High-performance networking technology optimized for low latency and high bandwidth in AI/HPC clusters.

Stopping or evicting a running job so resources can be reallocated to higher-priority workloads.

Memory bandwidth is the rate at which data can move between the GPU and its memory, often determining the ceiling of model performance.

Running a workload across multiple servers connected by a network fabric.

A Network Interface Card connects a server or GPU node to the network and determines its bandwidth and latency capabilities.

A single physical server containing CPUs, GPUs, memory, storage, and network interfaces.

A condition where available network bandwidth is less than what workloads would need at peak.

A distributed training approach where model parameters are centrally stored and updated.

Remote Direct Memory Access, allowing one machine to access another’s memory with minimal CPU overhead.

RDMA over Converged Ethernet, enabling RDMA-style performance on Ethernet networks.

Increasing performance or capacity by adding more nodes to the cluster.

Increasing performance by adding more GPUs within a single node (often via NVLink/NVSwitch).

Running a workload entirely within one server.

A Service Level Agreement that defines guaranteed performance, availability, and responsibilities.

A Service Level Indicator used to measure service performance.

A slow worker that bottlenecks a distributed job’s overall progress.

SXM and PCIe are two GPU form factors—SXM supports higher power and denser multi-GPU interconnects, while PCIe is modular and widely compatible.

Telemetry refers to real-time sensor data, such as temperature, power, and flow rates, used to monitor GPU clusters and facilities.

How GPUs, nodes, and switches are physically and logically interconnected.

A cluster that uses VMs and/or containers to isolate workloads and tenants.

A data center designed specifically for high-density GPU workloads and AI infrastructure.

A protective electrical device that interrupts power during faults.

A high-current conductor used for efficient power distribution.

Cooling equipment that removes heat from a fluid loop.

Hosting customer equipment in a third-party data center facility.

A Coolant Distribution Unit circulates and manages chilled liquid for liquid-cooled racks in a data center.

Data Center Infrastructure Management software monitors power, cooling, racks, and environmental conditions across the facility.

How much compute or power is installed per rack.

Liquid cooling delivered directly to CPUs or GPUs via cold plates.

A smaller data center located close to end users to reduce latency.

Cabling that transmits data as light for high-bandwidth links.

Equipment that provides longer-duration backup power when utility power fails.

Capturing and repurposing waste heat from IT systems.

A layout strategy separating hot exhaust air from cold intake air to improve cooling efficiency.

Very large-scale data center deployment with massive compute and power capacity.

Cooling method where servers are submerged in dielectric fluid.

The electrical power consumed by all equipment in a rack.

The time delay for data to travel from source to destination.

Cooling that uses liquid to remove heat more effectively than air.

Mean Time Between Failures, measuring reliability.

Mean Time To Repair, measuring serviceability.

Power Distribution Unit that delivers electrical power to rack equipment.

Power Usage Effectiveness is a metric that compares total facility power to IT power to measure data center efficiency.

A standardized enclosure that holds servers, switches, and power or cooling components.

A vertical measurement for rack equipment; 1U equals 1.75 inches.

Design approaches for fault tolerance using spare or duplicate systems.

A scalable two-tier data center network architecture.

Thermal Design Power is the maximum amount of heat a GPU is expected to generate and determines its cooling and power requirements.

The volume of data transferred per unit time.

Optical or electrical module enabling high-speed network links.

Uninterruptible Power Supply that provides short-term backup power.

Classification describing data center resiliency and expected availability.

A pricing or cost metric representing dollars per GPU per hour of usage.

A cost-efficiency metric representing dollars per unit of compute capability.

Capital expenditures for hardware, construction, and long-lived assets.

Emissions associated with electricity usage, usually expressed per kWh.

Compute output per watt of power consumed.

Inefficient resource use caused by poor job packing or allocation.

GPU utilization measures how much of a GPU’s compute capacity is actually being used, a key metric for cluster efficiency.

Operating expenditures such as power, staffing, and maintenance.