# Administration ### 1. Introduction to Base Command Manager (BCM) #### What is BCM? **Base Command Manager (BCM)** is a tool developed by NVIDIA that helps manage **GPU clusters**. Think of it like the **“control tower”** for a team of GPU-powered machines. It helps system administrators: - Add new machines (nodes) into the cluster - Monitor GPU usage and health - Control who can access what (user roles) - Coordinate with other tools like **Slurm** to run jobs on GPUs If you have **many servers with GPUs**, BCM helps you manage them **easily and efficiently**. #### What is a GPU Cluster? A **cluster** is a group of connected computers that work together. A **GPU cluster** is when those computers have **powerful NVIDIA GPUs** installed. Why clusters? AI workloads—especially deep learning—need **massive computation power**, and that often means **multiple GPUs across multiple machines**. #### Key Features of BCM (Explained Simply) | Feature | What It Means in Practice | | ----------------------------- | ------------------------------------------------------------------------------------ | | **Node registration** | You can add new GPU machines to your cluster and track their health. | | **GPU usage monitoring** | You can see how busy each GPU is (how much memory is used, how hot it is, etc.). | | **User/group access control** | You decide which users are allowed to do what (e.g., submit jobs, view logs). | | **Built-in tools** | BCM comes with other helpful tools like **Slurm** (for job scheduling) and **DCGM**. | #### How You Interact With BCM You can use **two main interfaces**: 1. **BCM CLI (Command-Line Interface)** - Type commands in a terminal - Example: `bcmsystem status` - Useful for scripting and automation 2. **BCM Web UI** - A webpage where you can see your cluster status visually - Easier for beginners #### Important Concepts ##### 1. `bcminit` This command is used to **initialize** a new node (server) and connect it to BCM. It sets up the necessary software and prepares the machine for management. ##### 2. Slurm Integration BCM doesn’t run AI jobs directly—it **passes them to Slurm**, which acts like the **job manager**. Think of it like this: BCM is the system manager. Slurm is the task manager. BCM makes sure Slurm knows where and how to run tasks. ##### 3. Health Checks BCM works with **DCGM** (Data Center GPU Manager) to make sure all GPUs are: - Running properly - Not overheating - Not reporting memory errors #### A Simple Example Scenario Let’s imagine your company has **5 GPU servers**. You install **BCM** on one server as the “controller,” then: - Use `bcminit` to connect the other 4 GPU servers - Now all 5 servers are visible in the **BCM Web UI** - You can check GPU usage in real time - You create user accounts: Alice (Admin), Bob (Data Scientist) - You install Slurm so Bob can submit AI training jobs to the GPU servers Congratulations! You now have a fully functioning **GPU cluster**. #### Real-World Use Cases of BCM - AI research labs managing training jobs across 20+ GPU nodes - Data centers optimizing GPU resource usage - MLOps engineers scheduling AI workloads in production #### Skills You Should Aim to Learn for BCM | Skill | Description | | -------------------- | ------------------------------------------------------ | | Install BCM | Set up BCM on a server and connect GPU nodes | | Use `bcminit` | Add new GPU nodes into the cluster | | Access the Web UI | View GPU status, memory, and node health visually | | Integrate with Slurm | Schedule jobs using Slurm after setting up the cluster | | Monitor with DCGM | Ensure GPU health (temperature, memory errors, etc.) | | Create user roles | Define permissions using BCM’s access control features | ### 2. Introduction to Slurm Workload Manager #### What is Slurm? **Slurm** stands for **Simple Linux Utility for Resource Management**. It is an **open-source job scheduler** used to manage and allocate **compute resources** (like GPUs and CPUs) in a **cluster environment**. Think of Slurm as the **“job dispatcher”** in your GPU cluster. When a user wants to run an AI training job, Slurm decides **where and when** that job should run. #### Why Do We Need Slurm? When you have multiple users and limited resources (say, 10 GPUs shared across a team), you need a **smart system** to: - Decide which job runs first - Make sure resources are used efficiently - Avoid conflicts (e.g., two users trying to use the same GPU) - Keep a history of all submitted jobs This is exactly what Slurm does. #### Key Use Cases of Slurm in AI Workloads | Use Case | Description | | ---------------------------- | ------------------------------------------------------------------ | | **AI/ML job scheduling** | Users can submit training/inference tasks to run on GPU nodes | | **Resource allocation** | Slurm can assign specific number of GPUs, CPUs, and memory per job | | **Job queue management** | Handles jobs waiting for available resources | | **Time and resource limits** | Prevents one user from using all resources forever | #### Important Components & Terminology Let’s break down some of the most important parts of Slurm: ##### 1. **slurmctld** (Slurm Controller Daemon) - This is the **central brain** of Slurm. - It manages job queues, resource allocation, and scheduling decisions. ##### 2. **slurmd** (Slurm Daemon) - Runs on every compute node. - It receives instructions from `slurmctld` and executes jobs. ##### 3. **slurm.conf** - The **main configuration file**. - It defines all nodes, partitions, limits, and user settings. ##### 4. **sbatch** - Command to **submit a job script** (usually a shell script with job instructions). ```bash sbatch train_model.sh ``` ##### 5. **squeue** - Shows all **jobs in the queue**, both running and waiting. ```bash squeue ``` ##### 6. **sinfo** - Displays **information about available compute nodes**. ```bash sinfo ``` #### How Slurm Works with BCM When you install **Base Command Manager (BCM)**, it includes Slurm as the **default scheduler**. Here’s the interaction: - BCM handles hardware, GPU drivers, and monitoring. - Slurm manages jobs and decides where and when to run them. So, for a complete AI infrastructure: - BCM = **manages the machines** - Slurm = **manages the jobs** #### A Beginner-Friendly Job Submission Workflow Let’s walk through a basic example of how a user runs a job using Slurm. ##### Step 1: Create a Job Script Create a file named `train_model.sh`: ```bash #!/bin/bash #SBATCH --job-name=ai_training #SBATCH --output=output.log #SBATCH --gres=gpu:1 #SBATCH --time=02:00:00 #SBATCH --mem=16G python train.py ``` Explanation: - `--job-name`: Name of the job - `--output`: Where to write the output logs - `--gres=gpu:1`: Request 1 GPU - `--time`: Maximum time the job can run - `--mem`: How much RAM to request ##### Step 2: Submit the Job ```bash sbatch train_model.sh ``` ##### Step 3: Check the Queue ```bash squeue ``` ##### Step 4: Monitor Output When the job starts, logs will be written to `output.log`. #### Tips for Beginners | Task | Command or Tool | | ------------------ | -------------------------------------------- | | Submit a job | `sbatch script.sh` | | Cancel a job | `scancel ` | | View the queue | `squeue` | | View node status | `sinfo` | | Debug a failed job | Check output log or use `sacct` for job info | #### Useful Job Options for AI Workloads | Option | Use Case | | --------------------- | --------------------------------------- | | `--gres=gpu:N` | Request N GPUs | | `--mem=XG` | Request memory (in GB) | | `--cpus-per-task=N` | Assign CPU cores for data preprocessing | | `--partition=NAME` | Choose a specific partition (queue) | | `--nodelist=HOSTNAME` | Run job on a specific machine | #### Skills You Should Aim to Learn for Slurm | Skill | Description | | --------------------------- | -------------------------------------------------------------- | | Install and configure Slurm | Set up `slurmctld`, `slurmd`, and `slurm.conf` | | Submit jobs | Use `sbatch` with proper job scripts | | Monitor jobs | Use `squeue`, `sinfo`, `sacct` to track and analyze job states | | Manage GPU resources | Set `--gres=gpu:N` and other limits to share GPUs efficiently | | Cancel or requeue jobs | Use `scancel`, `scontrol requeue` as needed | ### 3. Introduction to Fleet Command #### What is Fleet Command? **Fleet Command** is NVIDIA’s **cloud-based platform** designed to **deploy, manage, and monitor AI applications at the edge**. ##### What is the Edge? “**Edge**” means **computing done close to the data source**—for example: - A smart camera in a store - A robot in a factory - A medical device in a hospital - An AI server in a delivery truck Instead of sending all data to a central data center, **edge devices process data locally** for faster decisions. #### What Does Fleet Command Do? Fleet Command helps you: - **Register** remote edge devices (like Jetson, A100 servers, etc.) - **Deploy** AI software (like object detection models) - **Update** applications securely - **Monitor** system health and performance remotely - **Troubleshoot** without going on-site Think of Fleet Command as a **remote control system** for managing a “fleet” of AI devices spread across different locations. #### Why Use Fleet Command? | Without Fleet Command | With Fleet Command | | ------------------------------ | -------------------------------------------- | | Manually configure each device | Centralized dashboard to manage everything | | Travel to troubleshoot | Remote troubleshooting and logging | | Inconsistent deployments | Standardized container-based deployments | | Hard to monitor performance | Real-time monitoring of GPU health and usage | #### Fleet Command Workflow (Simplified) Here’s how a typical edge deployment works: ##### 1. **Register an Edge System** - You install a **Fleet Command agent** on the remote device. - It connects to the Fleet Command dashboard via the cloud. ##### 2. **Create an Application** - You define an application using **containers**. - You can use NVIDIA’s **NGC containers**, or build your own (using Docker). ##### 3. **Deploy the Application** - From the Fleet Command web interface, you select: - The device(s) you want to deploy to - The container to run - Fleet Command **remotely installs and starts** the app. ##### 4. **Monitor the System** - GPU temperature, usage, health, and logs are visible on the dashboard. - You can restart or stop applications anytime. #### Example: Deploying a Smart Camera AI System Let’s say a retail company wants to use AI cameras in 100 stores to count customers and monitor stock. With Fleet Command: - All 100 stores register their **Jetson devices** via a secure token. - The AI model is packaged into a Docker container. - The company deploys the container to all devices in one click. - If any camera fails, the support team can **see logs remotely** and restart the app—without visiting the store. #### Key Features | Feature | What It Means | | ------------------------ | -------------------------------------------------------------- | | **Secure provisioning** | Devices are added using **secure keys** and authentication | | **Remote deployment** | You can install/update software on devices **from anywhere** | | **Container support** | Applications run in **Docker containers** | | **Monitoring & logging** | GPU metrics, system status, logs—all in one dashboard | | **RBAC support** | Different users (admins, operators) have different permissions | #### Important Concepts ##### 1. **Edge Nodes** - A physical device (Jetson, EGX, etc.) registered into Fleet Command - These are the “workers” that run AI applications ##### 2. **Container Lifecycle** - Applications are deployed in containers - You can start, stop, restart, and update containers anytime ##### 3. **RBAC (Role-Based Access Control)** - Admins: Full access to all resources - Operators: Can deploy and monitor, but not change configs - Viewers: Read-only access to dashboards #### Skills You Should Learn for Fleet Command | Skill | Description | | -------------------------------- | ----------------------------------------------------------------- | | Register edge systems | Add new devices securely into Fleet Command | | Build containerized applications | Use Docker to package AI models and inference services | | Deploy containers remotely | Use the dashboard to launch apps on remote edge nodes | | Monitor GPU and system status | View health, usage, and errors in real-time | | Use RBAC | Define roles and responsibilities for different team members | | Troubleshoot remotely | Read logs, restart containers, fix issues without physical access | ### 4. Introduction to MIG (Multi-Instance GPU) #### What is MIG? **MIG** stands for **Multi-Instance GPU**. It is a **hardware-based virtualization feature** available on **NVIDIA Ampere (A100) and newer GPUs** that allows a **single physical GPU** to be split into **multiple isolated GPU instances**, called **MIG instances**. #### Why Use MIG? Without MIG: - One user might use the whole GPU, even if their task only needs a small portion. - Multiple users sharing the same GPU might interfere with each other’s workloads. With MIG: - A single GPU can be **divided into smaller "slices"**. - Each slice is **isolated**, with its own memory, cores, and cache. - Different jobs (from different users or containers) can run at the same time **without conflict**. #### Key Benefits of MIG | Benefit | Description | | ----------------------- | --------------------------------------------------------------------- | | **Isolation** | Each MIG instance runs independently, like its own mini-GPU | | **Resource Efficiency** | Allows small or medium jobs to run without wasting full-GPU resources | | **Multi-Tenancy** | Supports multiple users or apps on a single GPU | | **Improved Security** | Processes cannot access each other’s memory or compute | #### MIG Profiles (GPU "Slices") MIG profiles define how a GPU is split. For example, on an **NVIDIA A100 40GB GPU**, you can choose: | MIG Profile | Compute Cores | Memory | Description | | ---------------- | ------------- | ------ | ----------------------------- | | 1g.5gb | 1/7 of GPU | ~5 GB | Lightweight jobs or inference | | 2g.10gb | 2/7 of GPU | ~10 GB | Medium-sized training | | 3g.20gb | 3/7 of GPU | ~20 GB | Large inference or training | | 7g.40gb (no MIG) | 100% of GPU | 40 GB | Full access – not split | You choose a profile based on the **job’s size and memory needs**. #### MIG Setup Workflow (Step-by-Step) These steps are usually done on the command line using `nvidia-smi`. ##### Step 1: Enable MIG Mode Check whether MIG is enabled: ```bash nvidia-smi -L ``` To enable MIG: ```bash sudo nvidia-smi -mig 1 sudo reboot ``` ##### Step 2: Create MIG Instances Use `nvidia-smi mig` to list and create instances. Example: Create 3 instances of 1g.5gb profile: ```bash sudo nvidia-smi mig -cgi 0,1,2 -C ``` - `-cgi` means: “Create GPU Instances” - Each number represents a profile (e.g., 0 = 1g.5gb) ##### Step 3: View Active Instances ```bash nvidia-smi ``` You will see multiple **MIG instances** listed. Each one behaves like a separate GPU device. ##### Step 4: Assign MIG Devices to Containers or Jobs In Docker: ```bash docker run --gpus '"device=0"' nvidia/cuda:12.0-base nvidia-smi ``` In Kubernetes: - MIG devices are discovered by the **NVIDIA device plugin** - You can schedule pods onto MIG instances using device IDs #### Monitoring MIG with DCGM **DCGM** (Data Center GPU Manager) is fully compatible with MIG. You can: - Track individual MIG instance usage - Monitor temperature, memory, and errors - Set alerts and usage limits This is helpful when many users share one GPU. #### Common MIG Use Cases | Use Case | Example | | ---------------------------- | ------------------------------------------------------------------ | | Multi-user notebook platform | Give each user a 1g.5gb slice of the A100 GPU | | AI inference farm | Run multiple AI models on the same GPU simultaneously | | Cloud GPU service provider | Sell smaller "GPU units" to different customers securely | | Developer test environments | Let multiple developers test models on their own virtual GPU slice | #### Skills You Should Learn for MIG | Skill | Description | | -------------------------------- | ------------------------------------------------------------------- | | Enable MIG mode | Use `nvidia-smi` to switch the GPU to MIG-capable mode | | Create GPU and compute instances | Use `nvidia-smi mig -cgi` and profile numbers | | Assign MIG instances | Assign specific MIG devices to Docker containers or Kubernetes pods | | Monitor MIG performance | Use DCGM to track usage, errors, and health of each instance | | Match workloads to profiles | Choose correct MIG profile based on job memory and compute needs | ### 5. Introduction to DCGM (Data Center GPU Manager) #### What is DCGM? **DCGM** stands for **Data Center GPU Manager**. It’s a set of **tools, libraries, and APIs** provided by NVIDIA that helps system administrators: - **Monitor GPU performance and health** - **Detect hardware issues** - **Diagnose problems early** - **Track usage over time** Think of DCGM as a **“GPU health monitoring system”** for data centers and large-scale GPU deployments. #### Why Use DCGM? When managing **dozens or hundreds of GPUs**, it’s hard to know: - Which GPUs are running hot? - Are any GPUs producing memory errors? - Is any job using more power than allowed? - Which GPUs are underused? DCGM solves this by collecting **real-time telemetry** and providing **alerting** and **logging**. #### What Can DCGM Monitor? | Metric Type | Example Metrics | | ----------------------- | -------------------------------------------------------------- | | **Health** | Overall GPU health status, ECC memory errors, clock throttling | | **Performance** | GPU utilization, memory usage, power draw, SM occupancy | | **Temperature & Power** | GPU temperature, power limits, voltage | | **PCIe/NVLink** | Communication speed and error counters | | **MIG Instances** | Each instance’s performance, health, and error rate | #### DCGM Architecture ``` +-------------------------+ +-----------------------------+ | GPU Hardware Layer | ---> | DCGM Engine (on each node) | +-------------------------+ +-----------------------------+ | v +---------------------------+ | CLI / API / Exporters | +---------------------------+ ``` - The **DCGM Engine** runs on each GPU node - You interact with it using: - Command-line tools - API calls (C/C++/Python bindings) - **Prometheus exporters** for dashboards #### DCGM CLI Tool: `dcgmi` This is the **command-line interface** that allows administrators to interact with DCGM easily. ##### Common Commands ```bash dcgmi discovery -l ``` - Lists GPUs on the system ```bash dcgmi health -c ``` - Runs a health diagnostic test on all GPUs ```bash dcgmi stats --show ``` - Shows real-time stats like GPU utilization, memory use, temperature ```bash dcgmi diag ``` - Performs in-depth diagnostics to check hardware integrity #### Health Monitoring with DCGM DCGM can **automatically detect issues** like: - Memory errors (ECC errors) - Power/thermal throttling - Driver issues or crashes - MIG misconfigurations It assigns a **Health Status**: - **OK**: All good - **Warning**: Possible issues - **Critical**: Hardware likely failing or unstable #### Custom Alerts and Thresholds You can configure DCGM to **trigger alerts** when: - A GPU’s temperature exceeds 85°C - Power usage spikes - ECC errors happen more than once These alerts can be: - Logged locally - Sent to external systems (like Grafana dashboards) - Used to **automatically pause or migrate jobs** #### Integration with Prometheus and Grafana ##### Why integrate? To visualize GPU status on dashboards and track historical usage. ##### How it works: 1. **DCGM Exporter** runs on each node 2. It exposes metrics to Prometheus in the correct format 3. **Grafana** queries Prometheus and displays visual dashboards ##### Example Metrics: - `dcgm_gpu_utilization` - `dcgm_memory_used` - `dcgm_temperature` - `dcgm_power_usage` ##### Example Grafana Dashboard: - Per-node GPU usage - Cluster-wide temperature trends - Alerts when any GPU reaches thermal limit #### DCGM with Kubernetes or Slurm DCGM works well in environments like: - **Kubernetes**: With the NVIDIA GPU Operator, DCGM runs as a DaemonSet - **Slurm**: You can pull DCGM stats before scheduling jobs or after completion #### Skills You Should Learn for DCGM | Skill | Description | | --------------------------------- | ------------------------------------------------------------------------ | | Install and run DCGM | Set up DCGM Engine on GPU nodes | | Use `dcgmi` | Run diagnostics, check health, view metrics | | Configure alerts | Set thresholds for temperature, ECC, and other hardware faults | | Integrate with Grafana/Prometheus | Use exporters to visualize GPU metrics | | Monitor MIG instances | Track performance and errors per MIG slice | | Use with Kubernetes/Slurm | Integrate with modern cluster schedulers for unified GPU health tracking | ### 6. Role-Based Access Control (RBAC) #### What is RBAC? **Role-Based Access Control (RBAC)** is a security system that lets you **control who can do what** inside your infrastructure. Instead of giving full access to everyone, RBAC lets you: - Create **roles** (e.g., Admin, Operator, Viewer) - Assign **permissions** to each role - Assign **users** to those roles This protects your infrastructure from accidental or malicious actions. #### Why Is RBAC Important? | Without RBAC | With RBAC | | ------------------------------- | ------------------------------------- | | Anyone could shut down GPU jobs | Only Admins can stop/restart jobs | | Sensitive logs could be exposed | Viewers can’t access sensitive data | | Unsafe code could be deployed | Only Operators/Admins can deploy apps | RBAC is **critical** for: - **Multi-user systems** - **Edge environments** (e.g., Fleet Command) - **Kubernetes clusters** - **Cloud GPU infrastructure** #### Typical Roles in GPU/AI Environments | Role | Permissions | | ------------ | ------------------------------------------------------------------- | | **Admin** | Full access: add/remove nodes, assign roles, restart services | | **Operator** | Deploy applications, monitor systems, but **can’t change settings** | | **Viewer** | Read-only access to dashboards, logs, GPU status | #### RBAC in Different Platforms ##### In **Fleet Command**: - Web UI provides role-based login - Each user is assigned one of the 3 roles ##### In **BCM**: - Role permissions are configured during user account setup - You define: - Who can run jobs - Who can access node logs - Who can change Slurm or MIG settings ##### In **Kubernetes**: RBAC is **native** to Kubernetes. Example: A role that lets users **read** pods, but not delete them. ```yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: gpu-jobs name: viewer-role rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] ``` You can then assign that role using a **RoleBinding**. #### Skills You Should Learn for RBAC | Skill | Description | | --------------------------------- | ---------------------------------------------------------- | | Assign user roles in BCM or Fleet | Create user accounts and set their permissions | | Configure Kubernetes RBAC | Write and apply roles and bindings for GPU workload access | | Restrict sensitive actions | Allow only Admins to stop GPUs or change MIG setup | | Combine with monitoring tools | Limit who can see logs or usage metrics | ### 7. GPU Telemetry & Monitoring Integration #### What is GPU Telemetry? **Telemetry** means collecting real-time data from your system so you can: - Monitor performance - Detect problems - Make decisions based on usage patterns For GPUs, telemetry includes: - Utilization - Temperature - Power consumption - Memory usage - Error states (like ECC) #### Why GPU Monitoring Matters | Problem | What Monitoring Reveals | | ------------------- | ------------------------------------------------ | | GPUs overheating | See temperature rise and take action early | | Low GPU usage | Spot underused resources and optimize workloads | | Repeated ECC errors | Detect hardware issues and schedule replacements | | Power spikes | Identify power-hungry workloads | #### Monitoring Stack: **DCGM + Prometheus + Grafana** Here’s how they work together: ``` +----------------+ +----------------+ +------------------+ | DCGM Exporter| ---> | Prometheus | ---> | Grafana | +----------------+ +----------------+ +------------------+ (GPU data) (stores metrics) (visual dashboard) ``` ##### 1. DCGM Exporter - Installed on each GPU node - Collects GPU stats and exposes them to Prometheus ##### 2. Prometheus - Periodically scrapes GPU metrics from exporters - Stores time-series data - Can trigger alerts ##### 3. Grafana - Connects to Prometheus - Builds dashboards with charts, tables, gauges - Supports alert rules and notifications (email, Slack, etc.) #### Sample Metrics to Track | Metric Name | Meaning | | ---------------------- | ------------------------------ | | `dcgm_gpu_utilization` | Percentage of GPU in use | | `dcgm_memory_used` | GPU memory currently allocated | | `dcgm_temperature` | Current GPU temperature | | `dcgm_ecc_errors` | Memory error count | | `dcgm_power_usage` | Current power draw in watts | #### Kubernetes Integration If you're using Kubernetes with NVIDIA GPUs: - Use the **NVIDIA GPU Operator** to install: - DCGM exporter - Device plugin - Monitoring agents - You can also deploy the **kube-prometheus-stack** to get: - Pre-built dashboards - Node metrics - Cluster health alerts #### Skills You Should Learn for GPU Monitoring | Skill | Description | | ----------------------------- | ------------------------------------------------------- | | Install DCGM exporter | Expose GPU metrics on each node | | Deploy Prometheus and Grafana | Collect and visualize telemetry | | Create dashboards | Show temperature, usage, and errors in real-time | | Set alert rules | Get notified when GPUs overheat or show critical errors | | Use with Kubernetes or Slurm | Extend monitoring to job-level visibility | ### Final Summary: What You’ve Learned in "Administration" | Tool/Concept | Key Skills | | ------------------- | -------------------------------------------------------- | | **BCM** | Cluster management, node registration, Slurm integration | | **Slurm** | GPU job scheduling, partitions, queue management | | **Fleet Command** | Edge AI deployment, remote monitoring, app updates | | **MIG** | GPU slicing, isolation, configuration, and assignment | | **DCGM** | GPU health checks, metrics, diagnostics | | **RBAC** | User role control for safe and secure multi-user systems | | **Telemetry Tools** | Prometheus, Grafana, alerts, historical tracking | --- # Workload Management ### 1. Kubernetes for AI Workload Management #### Why Use Kubernetes for AI Workloads? **Kubernetes** (often shortened as **K8s**) is a powerful **open-source container orchestration system**. In AI environments, it helps: - **Deploy** and **manage** containerized ML/AI applications - **Scale** workloads automatically - Ensure **high availability** and **efficient use of GPUs** - Run jobs on **multi-node clusters** Kubernetes is **especially valuable for MLOps pipelines**, large-scale training, and inference systems. ##### Containers + Kubernetes = Scalable AI Before Kubernetes, people ran scripts manually or used tools like Docker Compose. But in large environments, you need to manage: - Dozens of models - Thousands of jobs - Dynamic GPU availability - Failures and restarts **Kubernetes automates all of this.** #### Key Kubernetes Concepts for AI ##### 1. **Pods & Nodes** | Term | Description | | -------- | ---------------------------------------------------------------------------------------------------------------- | | **Pod** | The **smallest unit** in Kubernetes; usually runs **one container** (e.g., a training job or inference service). | | **Node** | A **physical or virtual machine** (e.g., a GPU server) that runs Pods. Nodes can be GPU-capable or CPU-only. | Example: A pod might contain your `train.py` running on a Docker container. Kubernetes places this pod onto a node that has available GPU(s). ##### 2. **DaemonSets** A **DaemonSet** is a special controller that **runs a specific pod on every node**. Used for: - GPU monitoring tools like **DCGM exporter** - Logging agents - Node-level background processes Think of it as “auto-install this tool on every machine in the cluster.” ##### 3. **NVIDIA GPU Operator** Instead of installing GPU drivers and tools manually, **NVIDIA provides a Kubernetes Operator** to automate everything. The **NVIDIA GPU Operator** installs: - **GPU drivers** - **DCGM for GPU health monitoring** - **NVIDIA device plugin** (exposes GPU to pods) - **Container Runtime (e.g., nvidia-container-runtime)** It makes Kubernetes GPU-ready without manual setup. ##### 4. **CRI-O / containerd** These are **lightweight container runtimes**, alternatives to Docker. In modern Kubernetes clusters: - Docker is being **replaced** by **containerd** or **CRI-O** - NVIDIA’s GPU integration **works with all of them** You don’t need to know deep details, just know that **Kubernetes uses these runtimes to launch containers**. #### NVIDIA Device Plugin This plugin **connects NVIDIA GPUs to Kubernetes**. ##### What It Does: - Detects GPU hardware - Tells Kubernetes how many GPUs are available - Allows users to **request 1 or more GPUs per pod** ##### Example: Request 1 GPU in Pod YAML ```yaml resources: limits: nvidia.com/gpu: 1 ``` This ensures Kubernetes **schedules the pod on a node with at least 1 available GPU.** #### GPU Scheduling Strategy in Kubernetes Since AI workloads are **resource-hungry**, Kubernetes lets you **control exactly where and how** GPU jobs are scheduled. ##### Key Techniques: | Technique | What It Controls | | ------------------------------ | -------------------------------------------------------- | | **Node Selectors** | Only run on nodes with a GPU label (e.g., `gpu=true`) | | **Taints and Tolerations** | Prevent regular jobs from using GPU nodes unless allowed | | **Affinity Rules** | Co-locate jobs together (or spread them out) | | **Resource Requests & Limits** | Ensure fair usage of CPU, RAM, and GPUs | ##### Example: Pod Spec with Node Selector and GPU Limit ```yaml apiVersion: v1 kind: Pod metadata: name: ai-job spec: nodeSelector: accelerator: nvidia containers: - name: trainer image: myrepo/train:latest resources: limits: nvidia.com/gpu: 1 ``` - This pod **must run on a node labeled with `accelerator=nvidia`** - It will be **assigned exactly 1 GPU** ##### Beginner Tasks to Practice with Kubernetes | Task | How to Do It | | --------------------------------- | ----------------------------------------------------------------- | | Deploy a pod with GPU | Create a YAML file with `nvidia.com/gpu: 1` in `resources.limits` | | Label a node for GPU jobs | `kubectl label node my-node accelerator=nvidia` | | View node resources | `kubectl describe node ` | | Monitor pod status | `kubectl get pods`, `kubectl describe pod ` | | Install GPU Operator (using Helm) | `helm install nvidia-gpu-operator nvidia/gpu-operator` | ### 2. Slurm-Based Workload Management #### Why Use Slurm? **Slurm** (Simple Linux Utility for Resource Management) is a **job scheduler** used in **High-Performance Computing (HPC)** and **on-premises AI clusters**. It is especially common in environments where: - Fine-grained **control over GPU/CPU resources** is needed - **Batch jobs** (e.g., long AI training runs) are common - You want **predictable scheduling**, even without containers While Kubernetes excels at flexibility and containerization, **Slurm is often preferred when you need deterministic control**, especially in academic labs, national data centers, and tightly managed enterprise clusters. #### How Slurm Works (Simple Explanation) Imagine a queue at a coffee shop: - People (jobs) arrive and line up - The barista (Slurm) takes the next person in line - If the espresso machine (GPU) is busy, the next person waits That’s how **Slurm queues jobs and assigns hardware**. #### Slurm Job Lifecycle (Step-by-Step) ##### 1. Write a Job Script Create a Bash script to define: - How many GPUs and CPUs you want - How long your job can run - What commands to run Example: `train_model.sh` ```bash #!/bin/bash #SBATCH --job-name=training #SBATCH --output=output.log #SBATCH --gres=gpu:1 #SBATCH --mem=16G #SBATCH --time=04:00:00 python train.py ``` ##### 2. Submit the Job ```bash sbatch train_model.sh ``` Your job is added to the **queue** and waits until resources are available. ##### 3. Monitor the Queue ```bash squeue ``` See who is running, who is waiting, and which node each job is on. ##### 4. Check Job Logs Slurm writes **stdout and stderr** to the file you specify with `--output=...`. ```bash cat output.log ``` #### Important Slurm Commands | Command | Description | | --------- | ----------------------------------------- | | `sbatch` | Submit a job script | | `squeue` | View jobs in queue | | `scancel` | Cancel a job | | `sinfo` | Show node and partition status | | `sacct` | View historical job info (after they run) | #### Advanced Features in Slurm ##### 1. **QoS (Quality of Service)** - Controls job **priority**, **time limits**, and **resource limits** - For example, "student jobs" may have lower priority than "production jobs" ##### 2. **Partitions** - Logical grouping of nodes (like “queues”) - You can define: - A “GPU” partition for nodes with GPUs - A “debug” partition for short test jobs - Each partition can have its own limits and policies Example: ```bash #SBATCH --partition=gpu ``` ##### 3. **Job Preemption** - High-priority jobs can **interrupt lower-priority ones** - Useful when urgent tasks need to run immediately #### SlurmDBD (Slurm Database Daemon) - Used to **record job history**, usage stats, and billing data - Helps track: - Which users use the most GPUs - How much compute time each project consumes #### Real-World Example: Scheduling AI Jobs with Slurm A data science team wants to train 3 models: - Model A needs 2 GPUs for 4 hours - Model B needs 1 GPU for 8 hours - Model C is just a test and needs 10 minutes With Slurm: 1. Each team member writes a `sbatch` script 2. They submit jobs to the `gpu` partition 3. Slurm queues them and runs them as resources become available 4. Logs are saved and usage is recorded in SlurmDBD #### Beginner Tasks to Practice with Slurm | Task | Command or Action | | -------------------------- | -------------------------------------------------- | | Submit a GPU job | `sbatch train.sh` | | Request specific resources | Use `--gres=gpu:N`, `--mem=XXG`, `--cpus-per-task` | | Choose partition | `--partition=gpu` | | View jobs | `squeue`, `sacct` | | Cancel a job | `scancel ` | #### Summary: Kubernetes vs. Slurm | Feature | Kubernetes | Slurm | | -------------------- | -------------------- | ------------------------------ | | Container Support | Excellent (native) | Requires Docker/Singularity | | Job Scheduling Model | Dynamic (on-demand) | Batch (queued) | | GPU Awareness | Via device plugin | Built-in with `--gres=gpu:N` | | Multi-user Clusters | Supported | Supported | | Popular In | Cloud, DevOps, MLOps | HPC, research labs, on-prem AI | ### 3. Multi-Tenancy and Isolation #### What is Multi-Tenancy? **Multi-tenancy** means that **multiple users or teams share the same infrastructure**—like GPU clusters, Kubernetes clusters, or Slurm environments. It’s common in: - AI research groups - Universities - Enterprise AI platforms - Public or hybrid cloud environments Each “tenant” is like a separate user, team, or project. Good infrastructure ensures they can **co-exist without interfering** with each other. #### Why Isolation Is Important Without isolation, you might face: | Problem | Example | | ----------------------------- | ------------------------------------------------ | | One user uses all the GPUs | Slows down other users’ jobs | | User A crashes User B’s job | Shared memory or job conflicts | | Sensitive data is leaked | No secure boundaries between containers or users | | Logs and metrics are mixed up | Hard to debug who caused what | #### Techniques for Isolation in AI Infrastructure ##### 1. **MIG (Multi-Instance GPU)** Already covered in the **Administration** section, but here’s a recap: - MIG divides a **single physical GPU** into multiple **isolated instances** - Each instance has **dedicated memory, cache, compute cores** - Multiple users can run AI jobs **safely** on the same GPU **Example:** - 1 user gets a `1g.5gb` MIG slice - Another user gets a `2g.10gb` slice - Each runs separately, no interference ##### 2. **Kubernetes Namespaces** A **namespace** in Kubernetes is like a **virtual environment** or “folder” for resources. Each team or project can have: - Its own pods - Its own config - Its own resource quotas - Its own logs and metrics **Benefits:** - Separation of users and workloads - Better access control (via RBAC) - Quota enforcement (e.g., max 5 GPUs) **Create a namespace:** ```bash kubectl create namespace team-a ``` **Run a pod inside that namespace:** ```bash kubectl apply -f pod.yaml -n team-a ``` ##### 3. **Slurm Partitions** In Slurm, a **partition** is like a “queue” or “group of nodes.” You can assign specific nodes to specific users or jobs: - Partition `gpu-research`: 4 A100 nodes for research team - Partition `gpu-production`: 8 RTX 6000 nodes for production team **Control access using ACLs** (access control lists): ```bash PartitionName=gpu-research Nodes=node[01-04] AllowUsers=alice,bob ``` **Submit a job to that partition:** ```bash sbatch --partition=gpu-research script.sh ``` ##### 4. **Fair-Share Scheduling (Slurm)** Used when many users submit jobs, but some are more active than others. Slurm keeps track of **how much compute time each user has consumed** and **adjusts job priorities accordingly**. | User | Used GPU Hours | Job Priority | | ----- | -------------- | ------------ | | Alice | 0 | HIGH | | Bob | 100 | LOW | **Controlled by:** - `PriorityWeightFairshare` - `FairShare` value in Slurm configuration ##### 5. **Node Affinity and Taints (Kubernetes)** You can **isolate workloads** based on node labels. **Example:** - Label nodes for different teams: ```bash kubectl label node gpu-node-1 team=mlops ``` - Schedule pods only on “mlops” nodes: ```yaml nodeSelector: team: mlops ``` Also: - **Taints** prevent jobs from running on certain nodes unless they "tolerate" them - Used for **dedicated GPU nodes**, security boundaries, or debug workloads #### Real-World Use Case **Scenario:** Three teams (A, B, and C) share a Kubernetes cluster with 8 GPUs. | Goal | Strategy | | ----------------------------- | ------------------------------------- | | Team A can use 4 GPUs | Create namespace “team-a” + GPU quota | | Team B runs on separate nodes | Use nodeSelector + taints | | Team C shares GPUs via MIG | Assign MIG slices to their pods | Result: Each team runs independently without affecting each other. #### Skills You Should Learn for Multi-Tenant Isolation | Skill | Description | | ------------------------------------ | --------------------------------------------------------------- | | Use MIG | Allocate safe, isolated GPU slices to users | | Create Kubernetes namespaces | Separate workloads and configs by team or project | | Set resource quotas (K8s) | Prevent any user/team from consuming too many GPUs | | Configure Slurm partitions | Assign GPU nodes to specific users or groups | | Enable fair-share scheduling (Slurm) | Automatically adjust priorities based on usage history | | Use node selectors and taints | Pin workloads to specific GPU nodes or prevent job misplacement | ### 4. Monitoring & Logging #### Why Monitoring and Logging Matter In AI infrastructure, it's not enough to just run jobs. You also need to: - **Track performance**: Are GPUs being fully used? - **Catch failures early**: Are jobs crashing due to memory limits? - **Optimize resource usage**: Are some nodes overloaded? - **Audit and debug**: Who ran what, and when? Without proper monitoring and logging, you’re **flying blind**—especially in large, multi-user environments. #### Monitoring Tools (GPU-Focused) ##### 1. **Prometheus** - An **open-source monitoring system** - It **scrapes metrics** from services like DCGM (Data Center GPU Manager) - Stores time-series data (metrics over time) - Works with alert systems and Grafana dashboards **What It Monitors:** - GPU utilization - Memory usage - Temperature - Power draw - Error events (e.g., ECC errors) **How It Works:** - Each GPU node runs a **DCGM exporter** - Prometheus queries those exporters at regular intervals (e.g., every 15s) ##### 2. **Grafana** - A **dashboard tool** that connects to Prometheus - Displays data visually: graphs, gauges, heatmaps, alerts **Common Dashboards:** - Per-GPU usage over time - Job-by-job memory usage - Temperature heatmaps for racks of GPU servers - Alerts for failed jobs or hardware issues Grafana helps **operators, admins, and even users** see what’s happening at a glance. ##### 3. **Kubectl Logs** (Kubernetes) - Shows **stdout/stderr** logs from containerized AI jobs - Use it to debug crashes, errors, and training logs **Basic Command:** ```bash kubectl logs pod-name -n namespace ``` **Use Cases:** - Check why a training job failed - See accuracy/loss progress - Capture Python traceback for failed jobs ##### 4. **Slurm Logs** - Every Slurm job has an **output log file**, defined with `--output=...` - By default, logs include: - stdout - stderr - Job status (completed, failed, cancelled) **Advanced Logs:** - `/var/log/slurm/slurmctld.log`: Scheduler events - `/var/log/slurm/slurmd.log`: Node-level execution details **Command to query job history:** ```bash sacct -j ``` #### Key Metrics to Monitor (GPU + Job-Level) | Metric | Why It Matters | | ------------------------------ | -------------------------------------------------------------------------- | | **GPU Utilization** | Low usage? Maybe your model is bottlenecked on CPU or I/O | | **Memory Usage** | Frequent out-of-memory errors? Reduce batch size or adjust memory requests | | **Temperature** | Overheating GPUs may throttle performance or even shut down | | **Power Draw** | Unusually high power usage may signal runaway jobs or hardware failure | | **Job Duration** | Track how long each job runs—compare against expected durations | | **Container Start/Stop Times** | Slow starts could signal image pull issues or node pressure | | **Node Pressure** | Track CPU, RAM, disk usage—important for co-scheduled AI jobs | #### Best Practices for AI Monitoring | Practice | Benefit | | ---------------------------------------- | ------------------------------------------------------------- | | Deploy Prometheus + Grafana cluster-wide | Unified monitoring for all GPU nodes | | Set GPU temperature/power alerts | Detect early signs of overheating or stress | | Collect job-level logs | Debug model errors or resource issues | | Export metrics to centralized dashboard | Help admins and users understand performance | | Automate alerts | Email/Slack/Teams notifications on failure or critical events | #### Real-World Use Case **Scenario:** A training job is crashing after 2 hours with no clear error message. **Monitoring Workflow:** 1. Use `kubectl logs` (or Slurm job logs) to check for memory errors 2. Check Prometheus → see if GPU memory hit 100% 3. Grafana shows temperature spike before crash 4. Action: Reduce batch size + improve airflow in server rack #### Skills You Should Learn for Monitoring & Logging | Skill | Description | | ------------------------------- | ------------------------------------------------------------------ | | Use `kubectl logs` | View logs for AI training/inference jobs | | Set up DCGM exporter | Export GPU metrics for Prometheus | | Configure Prometheus scraping | Collect GPU stats cluster-wide | | Build Grafana dashboards | Visualize GPU usage, job health, and trends | | Use Slurm’s logging and `sacct` | Analyze completed job history and debug failed runs | | Alerting setup | Define conditions and notifications for GPU issues or job failures | ### 5. Resource Quotas and Limits (Kubernetes) #### Why Are Resource Quotas Important? In shared AI infrastructure, it’s common for **multiple teams** or users to: - Submit jobs at the same time - Request GPUs, CPUs, memory, etc. - Accidentally overuse shared resources Without quotas, one team could: - **Use all the GPUs** - Cause **denial of service** to others - Slow down or crash the system Resource quotas help you **ensure fairness** and **enforce usage limits per namespace or team**. #### Key Concepts ##### 1. **Namespace** Namespaces divide the cluster into **logical sections**, often used to group resources by: - Team (e.g., `team-a`, `team-b`) - Project - Environment (e.g., `dev`, `prod`) Each namespace can have **its own quota**. ##### 2. **ResourceQuota Object** This is a Kubernetes object that sets a **hard upper limit** on what a namespace can use. You can limit: - Number of pods - Total CPU or memory - Number of GPUs (`nvidia.com/gpu`) ##### Example: Limit a Namespace to 10 GPUs ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota namespace: team-a spec: hard: requests.nvidia.com/gpu: "10" ``` This means: - **Pods in the `team-a` namespace** can collectively **request up to 10 GPUs** - If one pod asks for 4 GPUs, and another for 7 → second one will be denied ##### Example: Limit CPU, Memory, and GPU ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-limits namespace: mlops-team spec: hard: requests.cpu: "40" requests.memory: "200Gi" requests.nvidia.com/gpu: "8" ``` This enforces: - Max 40 CPUs requested across all pods - Max 200 GiB of memory requested - Max 8 GPUs requested #### How Kubernetes Enforces These Limits Kubernetes tracks **total resources requested** in a namespace, not actual usage. So if you have this pod spec: ```yaml resources: requests: nvidia.com/gpu: 2 ``` Kubernetes: - Adds 2 GPUs to the total for the namespace - Rejects future pods if quota is exceeded That’s why it’s important to always define **resource requests** in your pod specs. ##### If You Don’t Set Resource Requests… - The quota won’t be enforced properly - Kubernetes might allow pods to overload nodes - You lose fine-grained control Always write this in your YAML: ```yaml resources: requests: nvidia.com/gpu: 1 limits: nvidia.com/gpu: 1 ``` #### Real-World Example **Scenario:** You have 3 teams sharing a Kubernetes cluster: - Team A: Needs up to 8 GPUs - Team B: Needs up to 4 GPUs - Team C: Development only, 2 GPUs max You: 1. Create 3 namespaces: `team-a`, `team-b`, `team-c` 2. Apply appropriate ResourceQuotas to each Now the cluster can safely host jobs without overloading any team. #### Skills You Should Learn for Resource Quotas | Skill | Description | | -------------------------------- | --------------------------------------------------- | | Create namespaces | Group workloads by team/project | | Apply ResourceQuota YAML | Enforce limits on GPU, CPU, memory | | Define resource requests in pods | Ensure pods count toward the quota system | | Audit resource usage | Use `kubectl describe quota` to check current usage | | Adjust quotas as needed | Increase or decrease based on project or team needs | ### 6. Job Lifecycle Automation #### Why Automate AI Jobs? Running AI jobs manually (e.g., `sbatch`, `kubectl apply`) works for one-off experiments. But in production or at scale, you need to: - Run the **same job daily, hourly, or on demand** - Chain steps: **preprocessing → training → evaluation → deployment** - Handle **failures, retries, logging, and monitoring** - Track inputs, outputs, and metrics This is where **workflow automation** tools come in. They help you build **repeatable, traceable, and scalable pipelines**. #### Popular Job Automation Tools for AI Workloads ##### 1. **Kubeflow Pipelines** **Kubeflow** is a Kubernetes-native MLOps platform. Its **Pipelines** module lets you build **end-to-end machine learning workflows**. | Feature | Benefit | | -------------------- | ----------------------------------------------- | | Step-based workflows | Break down jobs into modular components | | Reusability | Build once, run many times | | GPU scheduling | Native integration with Kubernetes + GPU plugin | | Visualization | View pipeline DAGs and metrics in a UI | **Example Workflow:** 1. Data Ingestion → 2. Feature Engineering → 3. Model Training → 4. Evaluation → 5. Deployment Each step is containerized and managed by Kubernetes. ##### 2. **Apache Airflow** Originally built for **data engineering**, Airflow is also widely used for: - **Scheduling training jobs** - **Managing ETL pipelines** - **Tracking long-running AI workflows** | Feature | Benefit | | ------------------------------ | -------------------------------------------- | | DAGs (Directed Acyclic Graphs) | Define tasks with dependencies | | Rich scheduling | Cron-like control (e.g., every night at 2am) | | Kubernetes integration | Launch tasks as Kubernetes pods | **Use Case:** - Train a model every hour - Trigger retraining when new data is uploaded to S3 - Send an alert if the job fails ##### 3. **Argo Workflows** Argo is a **Kubernetes-native workflow engine**. | Feature | Benefit | | ------------------------------- | ------------------------------------------ | | Container-native | Each step runs in its own Kubernetes pod | | GPU-aware | Fully supports `nvidia.com/gpu` resources | | Scalable and lightweight | Ideal for microservices or model inference | | YAML-based workflow definitions | Easy to store in Git for CI/CD workflows | **Example Use Case:** - Run distributed training on multiple pods - Automatically clean up intermediate files - Record run metadata and metrics #### Core Concepts in Workflow Automation | Concept | Description | | ------------------ | ------------------------------------------------------------------------------ | | **Workflow DAG** | A directed graph defining task order and dependencies | | **Step Container** | Each step in a workflow is a container that runs a defined script or command | | **Retry Policy** | Automatically retry a failed step X times before failing the workflow | | **Parameters** | Pass dynamic values (e.g., batch size, dataset path) to jobs | | **Artifacts** | Intermediate files (e.g., trained models, logs) saved and passed between steps | #### Benefits of Automating AI Workflows | Benefit | Explanation | | --------------------------- | ------------------------------------------------------------- | | **Reproducibility** | Always run the same code, same config | | **Scalability** | Can handle many jobs concurrently across multiple GPUs | | **Resilience** | Retry failed steps automatically | | **Traceability** | Track every step, log, metric, and artifact | | **Version Control + CI/CD** | Integrate with Git to enable production-grade MLOps pipelines | #### Real-World Use Case **Scenario:** A company trains a new NLP model every day at midnight using new data collected during the day. **Workflow:** 1. Step 1: Download new data from S3 2. Step 2: Clean and tokenize data 3. Step 3: Train Transformer model (requires 2 GPUs) 4. Step 4: Evaluate performance 5. Step 5: If accuracy > 90%, deploy to production Tools: - **Airflow** or **Argo** to automate scheduling and step execution - **Kubernetes + GPU plugin** to manage GPU allocation #### Skills You Should Learn for Job Automation | Skill | Description | | -------------------------------------- | ---------------------------------------------------------- | | Write workflow YAML (Argo/Kubeflow) | Define steps, containers, resources, dependencies | | Schedule jobs (Airflow) | Use DAGs and cron triggers to run workflows periodically | | Use `nvidia.com/gpu` in automated pods | Ensure GPU resources are allocated properly | | Monitor automated jobs | Use built-in dashboards and Prometheus/Grafana integration | | Handle failures gracefully | Add retries, alerting, logging to each step | ### 7. Real-World Scenarios You Must Understand These scenarios simulate the **types of problems and tasks you’ll face** in real AI operations roles and on the **NCP-AIO exam**. You should understand not just **what to do**, but **why it works** and how it interacts with Kubernetes, Slurm, GPU scheduling, monitoring, etc. #### Scenario 1: Run AI Training on Kubernetes with GPUs **Problem**: You need to deploy a containerized model training job that uses GPUs. ##### Solution: - Create a pod YAML: ```yaml resources: limits: nvidia.com/gpu: 2 ``` - Apply node selector to target GPU nodes: ```yaml nodeSelector: accelerator: nvidia ``` - Submit using `kubectl apply -f job.yaml` **Concepts involved**: - Kubernetes Pod spec - `nvidia.com/gpu` limits - GPU plugin - Node selectors #### Scenario 2: Monitor GPU Jobs Using Prometheus & Grafana **Problem**: You want real-time visibility into GPU usage per job. ##### Solution: - Deploy **DCGM Exporter** on each GPU node - Set up **Prometheus** to scrape metrics - Use **Grafana** to visualize: - GPU temperature - Memory usage - Power draw - ECC error counts **Concepts involved**: - DCGM - Prometheus scraping - Grafana dashboard setup - Node metrics + per-job statistics #### Scenario 3: Isolate Workloads from Multiple Teams **Problem**: Team A and Team B need to run jobs on the same cluster but must not interfere with each other. ##### Kubernetes-based Solution: - Create separate namespaces: `team-a`, `team-b` - Assign **ResourceQuotas** for each namespace - Use **RBAC** to restrict access - Optionally assign **MIG** slices to teams ##### Slurm-based Solution: - Define partitions: `team-a-partition`, `team-b-partition` - Assign users to each partition using `AllowUsers=` **Concepts involved**: - Namespaces - Resource quotas - MIG - Slurm partitions and access control #### Scenario 4: Limit One Team to 5 GPUs in Kubernetes **Problem**: You need to enforce a GPU usage limit for one team. ##### Solution: Create a `ResourceQuota` in their namespace: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: gpu-limit namespace: team-x spec: hard: requests.nvidia.com/gpu: "5" ``` **Concepts involved**: - ResourceQuota - Namespace scoping - GPU plugin resource requests #### Scenario 5: Handle Failed Job Due to Out-of-Memory (OOM) **Problem**: A training job fails after running for some time with a memory error. ##### Solution Steps: 1. Check logs: - `kubectl logs` or Slurm log output 2. Identify error: - "Killed: Out of Memory" 3. Fix: - Reduce batch size - Increase memory request: ```yaml resources: requests: memory: "32Gi" ``` **Concepts involved**: - Pod logs - Memory limits - Container lifecycle - Model optimization techniques #### Additional Practical Challenges to Be Familiar With | Situation | You Should Know... | | ------------------------------------- | ----------------------------------------------------------------- | | Job stuck in Pending state (K8s) | How to debug using `kubectl describe pod`, check GPU availability | | Slurm job runs but uses no GPU | How to request GPUs correctly with `--gres=gpu:1` | | Monitoring dashboard shows 0% GPU use | How to trace bottlenecks in CPU or I/O | | Kubernetes job fails silently | Where to find pod logs and error status | | One user using too many GPUs | How to apply quotas or fair-share limits | #### Final Summary: Skills You Must Master in **Workload Management** | Topic | Practical Skills | | ---------------------------- | -------------------------------------------------------------------------- | | **Kubernetes + GPU Plugin** | Write GPU-aware pod specs, use node selectors, apply limits | | **Slurm** | Submit jobs, manage partitions, enforce GPU usage policies | | **Monitoring** | Install DCGM exporters, configure Prometheus/Grafana, build dashboards | | **Isolation Techniques** | Use MIG, Kubernetes namespaces, and Slurm partitions to separate workloads | | **Scaling for Multi-Tenant** | Set up quotas, fair-share scheduling, and RBAC rules | | **Automation Tools** | Build workflows with Kubeflow, Argo, or Airflow for repeatable pipelines | --- # Installation and Deployment ### 1. System Prerequisites and Driver Installation #### Why This Matters Before running any AI workloads on a GPU, you must make sure the **operating system recognizes the GPU**, and the **software stack can talk to it properly**. That starts with: - Installing **NVIDIA GPU drivers** - Setting up a **container runtime** - Testing it using `nvidia-smi` or Docker containers This is the foundation. If this part fails, **nothing else will work**—including Kubernetes or Slurm. #### Step 1: Check Hardware Requirements Before installing anything: ##### Minimum Requirements: - A **NVIDIA GPU** (Ampere, Volta, or newer preferred for AI) - A **64-bit Linux OS** (Ubuntu 20.04/22.04 is most common) - **At least 8–16 GB RAM** and **sufficient disk space** - **Network connectivity** (for package installation and updates) To check if the system sees your GPU, run: ```bash lspci | grep -i nvidia ``` This should show a line like: ``` 01:00.0 VGA compatible controller: NVIDIA Corporation GA100 [A100] ``` #### Step 2: Install the NVIDIA GPU Driver This driver enables Linux to **communicate with the GPU hardware**. Without it, your system won’t know a GPU even exists. ##### Install Driver (Ubuntu) Open terminal and run: ```bash sudo apt update sudo apt install -y nvidia-driver-535 sudo reboot ``` - `nvidia-driver-535` is a **common version used with CUDA 12** - You can change the version depending on your CUDA/toolkit compatibility After reboot, test it with: ```bash nvidia-smi ``` You should see output like this: ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2 | | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | 0 A100-SXM4-40GB On | 00000000:3D:00.0 Off | 0 | +-----------------------------------------------------------------------------+ ``` If you see an error or `command not found`, the driver wasn’t installed correctly. #### Step 3: (Optional but Important) Install CUDA Toolkit If your AI workloads use **CUDA directly** (TensorFlow, PyTorch, or custom C++/CUDNN apps), you may also need the **CUDA Toolkit**. ##### Ways to Install: - **Runfile installer** from NVIDIA site (flexible, but manual) - **Package managers** (DEB/RPM via `apt` or `yum`) - **Containers** (most modern AI workflows use containers with CUDA pre-installed) For container workflows, the **CUDA Toolkit is often not required on the host**, only in the container image. #### Step 4: Install the NVIDIA Container Toolkit This allows **Docker** to run containers that can access the GPU. Without this, your AI containers won’t be able to see the GPU—even if `nvidia-smi` works on the host. ##### Setup Commands: Run these step by step: ```bash #Detect distribution distribution=$(. /etc/os-release;echo $ID$VERSION_ID) #Add NVIDIA’s package repo curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \ sudo tee /etc/apt/sources.list.d/nvidia-docker.list #Install the toolkit sudo apt update sudo apt install -y nvidia-container-toolkit #Restart Docker sudo systemctl restart docker ``` #### Step 5: Test GPU Container Access This is the final validation. ##### Run this command: ```bash docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi ``` You should see the same GPU stats output as you saw earlier. If you do, congratulations—**your GPU host is ready for container-based AI workloads**. #### Summary of Beginner Tasks You Should Master | Task | Command or Step | | -------------------------------- | -------------------------------------------------------- | | Verify GPU hardware | `lspci | | Install GPU driver | `sudo apt install nvidia-driver-535` + `nvidia-smi` | | Install NVIDIA container toolkit | Series of `curl` + `apt install` + `restart docker` | | Validate with test container | `docker run --gpus all nvidia/cuda:12.0-base nvidia-smi` | ### 2. Containerization and GPU Integration #### Why Containerization? **Containers** are lightweight environments that package: - An application (like your AI model) - Its dependencies (Python, libraries) - Runtime tools (CUDA, cuDNN) They allow you to: - Run AI workloads **consistently across machines** - Avoid "it works on my laptop but not on the server" - Deploy jobs at scale in **Kubernetes**, **Slurm**, or **Fleet Command** #### Why GPU Integration in Containers? By default, **Docker containers can’t access GPUs** unless you: 1. Install the **NVIDIA Container Toolkit** 2. Configure Docker to use the **NVIDIA runtime** This setup allows containers to run code on the GPU **just like the host system**. #### Docker vs. Podman (Brief Comparison) | Tool | Notes | | ---------- | --------------------------------------------------------------- | | **Docker** | Most commonly used; works well with NVIDIA toolkit | | **Podman** | Docker alternative; rootless by default; compatible with NVIDIA | For simplicity, **Docker** is recommended for beginners and is what most AI workloads still use. #### Installing the NVIDIA Container Toolkit (Step-by-Step) Already covered earlier, but here’s the logic behind each step: ##### 1. Identify OS Version (Why?) NVIDIA packages are **OS-specific**, so we detect which version of Ubuntu/Debian we’re using: ```bash distribution=$(. /etc/os-release;echo $ID$VERSION_ID) ``` ##### 2. Add NVIDIA’s Docker Repository This tells your system where to **find and download the container tools**: ```bash curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \ sudo tee /etc/apt/sources.list.d/nvidia-docker.list ``` ##### 3. Install Toolkit & Restart Docker ```bash sudo apt update sudo apt install -y nvidia-container-toolkit sudo systemctl restart docker ``` - `nvidia-container-toolkit` includes: - NVIDIA runtime - NVIDIA hooks to connect containers with GPUs #### Testing GPU Access in Containers ##### Test 1: See if Docker sees your GPU ```bash docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi ``` Expected output: ``` NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2 GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC A100-SXM4-40GB On | 00000000:17:00.0 Off | 0 ``` If it works, you now have: - Docker running - NVIDIA runtime active - A working GPU inside a container #### Optional: Set NVIDIA as Default Runtime To avoid having to specify `--gpus all` each time, modify Docker’s config: ```bash sudo nano /etc/docker/daemon.json ``` Add or update this section: ```json { "default-runtime": "nvidia", "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] } } } ``` Then restart Docker: ```bash sudo systemctl restart docker ``` Now all containers will assume GPU access unless otherwise specified. #### Using Docker Images with CUDA & AI Libraries Use official NVIDIA containers from **NGC (NVIDIA GPU Cloud)** or Docker Hub: | Image | Use Case | | ----------------------------------------- | ------------------------------ | | `nvidia/cuda:12.0-base` | Basic CUDA environment | | `nvcr.io/nvidia/pytorch:24.02-py3` | PyTorch with CUDA preinstalled | | `nvcr.io/nvidia/tensorflow:24.01-tf2-py3` | TensorFlow 2.x + CUDA | | `nvcr.io/nvidia/tritonserver:24.02-py3` | Inference with Triton | You’ll need to register for **NGC access** to pull from `nvcr.io`. #### Skills You Should Practice | Task | Command / Tool | | ----------------------- | -------------------------------------------------------- | | Run container with GPU | `docker run --gpus all nvidia/cuda:12.0-base nvidia-smi` | | Set default GPU runtime | Edit `/etc/docker/daemon.json` | | Pull CUDA container | `docker pull nvidia/cuda:12.0-base` | | Build custom AI image | `Dockerfile` + `docker build -t mymodel .` | | Debug GPU access issues | Use `nvidia-smi` in host and inside the container | ### 3. Kubernetes GPU Deployment #### Why Use Kubernetes for AI GPU Workloads? Kubernetes is the **standard platform** for running **containerized applications at scale**, including: - AI model training - Inference services - Scheduled batch jobs - Multi-user GPU clusters But **Kubernetes does not support GPUs natively**. You need to: - Install GPU drivers - Expose GPU hardware to Kubernetes - Set up monitoring tools That’s where the **NVIDIA GPU Operator** and supporting components come in. #### Step-by-Step: GPU Enablement in Kubernetes ##### Step 1: Install a Kubernetes Cluster You can use any of the following to set up a basic cluster: | Method | Use Case | | -------------- | -------------------------------- | | `kubeadm` | Production or large-scale setups | | `microk8s` | Lightweight, great for testing | | `minikube` | Local testing only | | `RKE` or `k3s` | Lightweight Kubernetes distros | For multi-node production environments, **kubeadm** is recommended. Example (single-node setup with kubeadm): ```bash sudo apt update sudo apt install -y kubelet kubeadm kubectl sudo kubeadm init --pod-network-cidr=10.244.0.0/16 ``` Apply a CNI (like Flannel or Calico), and then your cluster is ready. ##### Step 2: Install the NVIDIA GPU Operator The **GPU Operator** automates: - GPU driver installation - DCGM setup - Device plugin deployment - GPU monitoring tools (exporters, collectors) You can install it using **Helm** or `kubectl`. ###### Prerequisites: - Helm installed - NVIDIA Container Toolkit installed on nodes - `nvidia-smi` working on nodes ###### Install with Helm ```bash helm repo add nvidia https://nvidia.github.io/gpu-operator helm repo update helm install --wait \ --generate-name \ -n gpu-operator \ --create-namespace \ nvidia/gpu-operator ``` This creates a new namespace `gpu-operator` and deploys: | Component | Purpose | | -------------------- | --------------------------------------- | | **Driver container** | Installs GPU drivers inside a container | | **Device plugin** | Exposes GPUs to Kubernetes | | **DCGM exporter** | Sends GPU metrics to Prometheus | | **Validator** | Verifies if the node is fully GPU-ready | ##### Step 3: Check GPU Node Readiness Check the `nvidia.com/gpu` resource is available: ```bash kubectl get nodes -o json | jq '.items[].status.allocatable' ``` Or: ```bash kubectl describe node ``` You should see: ```yaml Allocatable: nvidia.com/gpu: 1 ``` This means your GPU is now **visible to Kubernetes**! ##### Step 4: Run a GPU-Powered Pod Example pod spec: ```yaml apiVersion: v1 kind: Pod metadata: name: gpu-test spec: containers: - name: nvidia-container image: nvidia/cuda:12.0-base command: ["nvidia-smi"] resources: limits: nvidia.com/gpu: 1 ``` Deploy it with: ```bash kubectl apply -f gpu-test.yaml ``` Check output with: ```bash kubectl logs gpu-test ``` You should see the same `nvidia-smi` output from your host system. #### Required GPU Components in Kubernetes | Component | Role | | ---------------------------- | ----------------------------------------------------- | | **nvidia-device-plugin** | Reports GPU hardware to the scheduler | | **nvidia-container-runtime** | Allows GPU access in Docker-based containers | | **dcgm-exporter** | Sends GPU telemetry to Prometheus | | **NVIDIA driver container** | Installs the correct GPU driver version automatically | All of these are installed **automatically by the GPU Operator**. #### Troubleshooting Tips | Problem | Fix | | ----------------------------------- | ------------------------------------------------------------- | | Pod stuck in `Pending` | Node may not have available GPUs or device plugin not running | | `nvidia-smi` fails inside container | Check if driver is correctly installed by operator | | No GPU resource on node | Restart operator pods or revalidate node readiness | | Metrics not appearing in Prometheus | Ensure `dcgm-exporter` is running and connected | #### Beginner Tasks You Should Try | Task | Command / File | | ----------------------- | --------------------------------------------- | | Install GPU Operator | `helm install nvidia/gpu-operator` | | Deploy GPU pod | Use `resources.limits.nvidia.com/gpu` in YAML | | Check GPU availability | `kubectl describe node` | | View container logs | `kubectl logs pod-name` | | Watch GPU plugin status | `kubectl get pods -n gpu-operator` | ### 4. Base Command Manager (BCM) Setup #### What is Base Command Manager (BCM)? **BCM** is NVIDIA’s enterprise tool for managing **AI clusters** equipped with NVIDIA GPUs. It provides a centralized way to: - **Configure and monitor GPU nodes** - **Submit, track, and manage AI workloads** - **Integrate with Slurm** for job scheduling - Visualize **GPU health**, **utilization**, and **job status** via **CLI or Web UI** BCM is ideal for on-premises GPU clusters in: - Research centers - Data centers - Enterprise AI labs #### Key Features of BCM | Feature | Description | | -------------------- | ------------------------------------------------- | | Node registration | Add and manage GPU-enabled servers in the cluster | | Slurm integration | Use Slurm to queue and dispatch AI jobs | | GPU monitoring | Track health, utilization, and errors (via DCGM) | | Access control | Assign users, roles, and resource quotas | | Web UI & CLI support | Full management experience from either interface | #### BCM Setup Workflow ##### Step 1: **Provision Host OS** Before you install BCM, you must: - Use a supported OS: **Ubuntu 20.04+, RHEL 8+, or CentOS 8+** - Ensure **NVIDIA GPU drivers** are installed - Confirm `nvidia-smi` works - Install container runtime (Docker) Example: ```bash sudo apt update sudo apt install -y nvidia-driver-535 docker.io nvidia-smi ``` ##### Step 2: **Install BCM Agent** The **BCM agent** is installed on every node (head + worker). You get the agent package from NVIDIA via: - Official ISO or installation script - Enterprise portal or DGX system image (if you have a DGX server) Run the installer on each node. ##### Step 3: **Initialize BCM on Each Node** On the head node: ```bash bcminit ``` This command: - Configures BCM services - Registers the host in the BCM controller - Sets up directories and default configs On worker nodes: ```bash bcminit --join --controller-ip ``` This joins the node to the cluster. ##### Step 4: **Access BCM Web UI** By default, BCM runs a web interface on port `8443` or `443`. Access it at: ``` https://:8443 ``` You can: - View all GPU nodes - Check usage and health - Submit jobs (through Slurm) - Add/remove users ##### Step 5: **Integrate Slurm** Slurm is included with BCM and is the primary scheduler. Check Slurm status: ```bash sinfo squeue ``` You can now submit AI jobs using: ```bash sbatch train_model.sh ``` Where `train_model.sh` is your job script, like: ```bash #!/bin/bash #SBATCH --job-name=training #SBATCH --gres=gpu:1 python train.py ``` ##### Step 6: **Validate the Cluster** To check if everything is working: ```bash bcmsystem status # Checks BCM services sinfo # Shows node availability squeue # Shows job queue nvidia-smi # Confirms GPU visibility ``` Also, open the **Web UI** and confirm: - Nodes are listed - GPUs are shown with health data - Slurm jobs appear in the interface #### BCM with MIG and DCGM BCM supports: - **MIG-based GPU slicing** You can assign specific MIG instances per job or user. - **DCGM integration** GPU telemetry is shown in the UI: - Temperatures - ECC errors - Utilization - Power draw #### BCM User Roles and Access Control In multi-user environments: - **Admins** manage infrastructure and configs - **Operators** manage jobs and monitor resources - **Users** submit jobs, view logs, and access datasets All permissions can be set via: - Web UI → User Settings - `bcmusers` CLI tools #### Skills You Should Learn for BCM | Skill | Description | | ------------------------------ | ---------------------------------------------------------------- | | Install BCM and register nodes | Use `bcminit` to configure both head and worker nodes | | Access and use the Web UI | Manage cluster health, users, jobs visually | | Use Slurm for job scheduling | Submit jobs using `sbatch`, monitor with `squeue`, `sacct` | | Monitor GPU performance | View GPU stats via UI (DCGM), CLI (`nvidia-smi`), and Slurm logs | | Manage users and roles | Create and configure roles for multi-tenant GPU clusters | ### 5. DOCA – Data Center-on-a-Chip Architecture #### What is DOCA? **DOCA** is NVIDIA’s software framework designed to run on **BlueField DPUs (Data Processing Units)**. It provides: - High-performance **networking acceleration** - **Security enforcement** (Zero Trust security at the infrastructure level) - **Data path offloading** from the CPU DOCA is **not about running AI models directly**, but about **optimizing and securing AI infrastructure**, especially when deploying across **high-performance edge or data center environments**. #### Key Components of DOCA | Component | Description | | ----------------- | ----------------------------------------------------------------------------- | | **DOCA SDK** | For developers to build custom apps that run on DPUs | | **DOCA Services** | Pre-built containerized apps for networking, security, and storage offload | | **DOCA Runtime** | Runtime engine for deploying and running DOCA applications | | **BlueField DPU** | The hardware platform that runs DOCA (like a smart NIC or infrastructure CPU) | #### Why DOCA Matters in AI Deployments | Use Case | How DOCA Helps | | -------------------------------------------- | ----------------------------------------------------------------- | | Secure inference at the edge | Enforce network isolation and deep packet inspection | | Large-scale AI model deployment | Offload storage/network processing from host CPUs | | Data pipeline acceleration (e.g. NVMe, PCIe) | Boost performance of data movement to/from GPU compute nodes | | Regulatory compliance (Zero Trust) | Ensure infrastructure-level trust, encryption, and access control | #### Example DOCA Use Cases in AI Infrastructure ##### 1. Secure AI Model Inference at the Edge - Imagine a hospital using AI models for **image classification** on patient data. - DOCA can: - Inspect network packets on the DPU before they reach the CPU or GPU - Run lightweight inference at the DPU level (e.g., for preliminary triage) - Ensure only encrypted traffic is allowed This **reduces latency**, **improves privacy**, and **offloads work** from central nodes. ##### 2. High-Speed Data Ingestion for Training - In data centers, AI workloads often pull data from: - NVMe over Fabrics - RDMA-enabled storage - Streaming video from remote sensors - DOCA can: - Accelerate this I/O pipeline directly on the DPU - Filter data packets to reduce CPU/GPU load - Preprocess or cache data closer to the GPU node #### How DOCA Applications are Deployed Most DOCA services are **packaged as containers** and run directly on the **BlueField DPU**, separate from the host CPU. You interact with DOCA via: ##### DOCA CLI: Used to manage DPU configuration and services. ```bash doca_app_manager list doca_service_control start ``` ##### NVIDIA NGC Registry: Many DOCA containers are hosted here. To deploy: 1. Use Fleet Command or container management system 2. Start services with `systemctl` or `docker` on BlueField OS 3. Connect to host applications or monitoring tools (like DCGM) #### Important DOCA Concepts You Should Know | Term | Meaning | | ----------------------- | ----------------------------------------------------------------------------- | | **SmartNIC** | Network Interface Card with built-in compute (i.e., BlueField) | | **Zero Trust Security** | All connections must be authenticated, authorized, and encrypted | | **PCIe Filtering** | Block or redirect data moving through PCIe bus using policy | | **Telemetry Offload** | Stream system monitoring data from DPU directly to remote observability tools | #### Beginner Tasks to Practice with DOCA (if access to BlueField available) | Task | How to Do It | | ---------------------------------- | ------------------------------------------------------ | | Access DOCA runtime on BlueField | SSH into DPU OS or serial console | | List DOCA containers or services | Use `docker ps` or `doca_app_manager list` | | Enable a DOCA service | `systemctl start doca-telemetry.service` or equivalent | | Monitor performance | Use NVIDIA NIM or Fleet Command monitoring | | Integrate with host GPU monitoring | Stream DPU telemetry alongside DCGM metrics | #### Summary: Why DOCA Matters in Installation & Deployment | Role in AI Infrastructure | Value It Brings | | -------------------------------------- | ------------------------------------------------- | | **Data security at the edge** | Keeps inference data private and compliant | | **Network acceleration** | Offloads TCP/IP stack, reduces CPU/GPU contention | | **PCIe and storage path optimization** | Speeds up data loading to GPUs for training | | **Telemetry collection and isolation** | Improves observability and system resilience | ### 6. Fleet Command Deployment #### What is Fleet Command? **Fleet Command** is NVIDIA’s **cloud-based management platform** for deploying and operating AI applications at the **edge**. It allows you to: - Manage fleets of **remote edge devices** (e.g., Jetson, A100, BlueField systems) - **Remotely deploy AI containers** (from NGC or custom registries) - Monitor **telemetry**, **health**, and **logs** - Perform **secure updates** and **troubleshooting** Ideal for industries like **retail, healthcare, manufacturing**, and **logistics** that need **remote AI processing** close to data sources. #### Edge Deployment Architecture Fleet Command consists of two key layers: | Layer | Function | | ----------------------- | --------------------------------------------------------------- | | **Cloud Control Plane** | Hosted by NVIDIA. Manages devices, applications, and monitoring | | **Edge Nodes** | The devices (e.g. Jetson, GPU servers) running the workloads | Edge nodes **pull workloads from the cloud** and **push logs and telemetry back**. #### Supported Hardware Platforms | Device/Server Type | Example Models | | ------------------------ | --------------------------------------- | | NVIDIA Jetson | Xavier NX, AGX Orin | | NVIDIA-Certified Servers | With A100, L40, H100, etc. | | BlueField DPUs | Combined with DOCA for secure inference | | OEM Edge Devices | Integrated with NVIDIA GPUs | #### Typical Fleet Command Use Cases | Industry | Example Use Case | | ------------ | --------------------------------------------------- | | Retail | Smart checkout, customer analytics | | Healthcare | Medical image inference at clinics | | Logistics | Real-time video analytics for safety and compliance | | Smart Cities | Edge traffic monitoring and public safety systems | #### Fleet Command Edge Deployment Process ##### Step 1: Register Device - Log into the **Fleet Command portal** on NVIDIA LaunchPad or enterprise dashboard. - Register a new device by: - Generating a **secure registration token** - Downloading the **Edge Node Installer** On the edge system: ```bash sudo bash edge-node-installer.sh --token ``` This: - Installs core services - Configures networking and security keys - Connects device to the Fleet Command control plane ##### Step 2: Configure Applications You can: - Use **prebuilt containers from NGC** - Upload **your own container images** (from private registries) - Define **deployment parameters**, such as: - Resource limits - Volume mounts - Environment variables ##### Step 3: Deploy Applications In the Web UI: - Select the device - Choose the app/container - Click **Deploy** Fleet Command pulls the container image, launches it on the device, and: - Starts the container with NVIDIA GPU access - Monitors logs and performance - Reports back to the UI You can **stop, update, or redeploy** at any time. ##### Step 4: Monitor and Troubleshoot You can view: - **System logs** (journal logs, container logs) - **Application logs** - **Device health**: CPU, memory, GPU, temperature - **Connectivity status** If a node fails or loses connection: - You get an alert in the dashboard - You can re-register or reset remotely #### Security Features in Fleet Command | Security Mechanism | Purpose | | -------------------------------- | ----------------------------------------------- | | Secure bootstrapping | TLS-authenticated registration process | | Remote software updates | Signatures and rollback support | | Isolated container runtime | Applications run in secure containers | | Role-Based Access Control (RBAC) | Define who can deploy, monitor, and access logs | Fleet Command is designed with **Zero Trust principles**—critical for edge deployments in regulated industries. #### Beginner Tasks to Practice (if you access LaunchPad or Fleet Command sandbox) | Task | How to Do It | | ---------------------------- | -------------------------------------------------- | | Register an edge device | Use `edge-node-installer.sh` with secure token | | Deploy an AI container | Select image, define parameters, deploy via Web UI | | View logs | Go to Logs tab for the device or application | | Monitor health | View resource usage and status in the dashboard | | Stop or restart applications | Click “Stop” or “Restart” in the App control panel | #### Summary: What You Gain with Fleet Command | Feature | Value | | --------------------------- | ----------------------------------------------------------- | | **Centralized control** | Manage all edge AI nodes from one dashboard | | **Remote deployment** | No physical access required to update or manage systems | | **Real-time monitoring** | Know immediately if a workload or device has an issue | | **Scalability** | Manage tens, hundreds, or thousands of edge nodes | | **Security and compliance** | Protect sensitive data and models in untrusted environments | ### 7. Magnum IO – Multi-GPU, Multi-Node I/O Acceleration #### What is Magnum IO? **Magnum IO** is a suite of **software libraries, tools, and drivers** from NVIDIA designed to **optimize I/O (input/output) operations** for: - **Multi-GPU** workloads - **Multi-node** training clusters - **High-performance AI and HPC applications** It enables GPUs to communicate faster with each other and with storage and networking systems—**eliminating bottlenecks** in distributed training or inference. #### Why Magnum IO Matters for AI Workloads | Challenge | How Magnum IO Helps | | -------------------------------------- | --------------------------------------------------- | | Multi-GPU communication latency | Uses **NVLink/NVSwitch** with **NCCL**, **UCX** | | Poor I/O performance in large clusters | Accelerates data movement via **GPUDirect** | | CPU bottlenecks in communication stack | Offloads I/O using **DPU + GPUDirect RDMA** | | Inefficient distributed training | Ensures **synchronized model updates** via **NCCL** | #### Key Components of Magnum IO | Component | Description | | -------------------------------------------------- | -------------------------------------------------------------- | | **NCCL (NVIDIA Collective Communication Library)** | Handles GPU-to-GPU data movement across nodes | | **UCX (Unified Communication X)** | Framework that abstracts different transport methods | | **UCC (Unified Collective Communication)** | Layer that sits above UCX for collective ops | | **GPUDirect RDMA** | Enables network cards to communicate directly with GPU memory | | **GPUDirect Storage** | Enables storage devices to read/write from GPU memory directly | #### Typical Use Case: Distributed AI Training Let’s say you're training a ResNet model on 16 A100 GPUs spread across 4 servers. Without Magnum IO: - Each GPU sends gradients through the CPU and NIC to other GPUs - Data sync is **slow and CPU-bound** With Magnum IO: - Gradients are sent **directly GPU-to-GPU using NVLink + NCCL** - **GPUDirect RDMA** handles networking bypassing CPU - Result: **Faster convergence** and **lower training time** #### How It Works (Simplified Diagram) ``` [GPU 1]───NVLink───[GPU 2] │ │ └─GPUDirect──NIC───┘ │ Ethernet/IB │ [Other Node] ``` - GPUs communicate over **NVLink/NVSwitch** inside the node - Data goes through NICs directly using **GPUDirect** - **NCCL** handles collective operations like `AllReduce`, `Broadcast` #### How to Enable and Use Magnum IO 1. **Use NVIDIA NGC Containers** (e.g., PyTorch with NCCL preinstalled): ```bash docker run --gpus all nvcr.io/nvidia/pytorch:24.02-py3 ``` 2. **Enable GPUDirect RDMA** in the kernel and driver - Kernel modules like `nvidia_peermem` must be loaded - NIC (e.g., Mellanox) must support RDMA 3. **Use NCCL backend in training code**: - PyTorch: ```python dist.init_process_group(backend='nccl') ``` - TensorFlow: Uses `horovod` or `tf.distribute` under the hood with NCCL 4. **Ensure correct topology**: - Run `nvidia-smi topo -m` to see NVLink/NVSwitch layout 5. **Monitor performance**: - Use `nvprof`, `nsys`, or DCGM metrics - Watch NCCL logs for communication efficiency #### Best Practices for Magnum IO Clusters | Practice | Description | | ------------------------------------- | -------------------------------------------------------------- | | Use NVSwitch/NVLink where available | Enables full bandwidth between GPUs | | Use Mellanox NICs with RDMA | Required for GPUDirect RDMA | | Align process placement with topology | Use tools like `mpirun --map-by ppr:...` to colocate processes | | Use NGC containers with NCCL | Preconfigured and optimized for distributed training | | Enable `nv_peer_mem` on all nodes | Required for GPUDirect to work correctly | #### Summary: Why Magnum IO Is Critical | Benefit | Result | | -------------------------------- | ------------------------------------------------------------ | | Fast multi-GPU communication | Speeds up gradient exchange and model updates | | CPU offloading | More resources available for preprocessing or other tasks | | Optimized I/O paths | Reduces training time and improves throughput | | Compatibility with AI frameworks | Works with PyTorch, TensorFlow, Horovod, etc. | | Scalable to hundreds of nodes | Suitable for supercomputers and hyperscale AI infrastructure | ### 8. Storage Considerations & Deployment Automation #### Why Storage Matters in AI Workloads AI workloads are **data-intensive**. Poor storage architecture can cause: - **Slow training speeds** - **Data loading bottlenecks** - **Underutilized GPUs** To avoid this, you must: - Use **high-throughput file systems** - Optimize **I/O paths** - Enable **RDMA** where possible #### Storage Considerations for AI ##### 1. Use Parallel File Systems | File System | Notes | | ----------------------------- | -------------------------------------------- | | **Lustre** | High-performance, widely used in HPC | | **BeeGFS** | Easy to scale, optimized for mixed workloads | | **IBM Spectrum Scale (GPFS)** | Very scalable, enterprise-ready | These systems: - Split large files across multiple storage servers - Support **many concurrent readers/writers** (perfect for multi-GPU training) ##### 2. Use RDMA (Remote Direct Memory Access) RDMA allows storage/network adapters to: - Transfer data **directly between memory regions** - Bypass CPU - Lower latency and increase bandwidth Especially helpful when using **GPUDirect Storage** in large AI training clusters ##### 3. Local SSDs and Data Caching - Use **NVMe SSDs** for local caching or high-speed scratch space - Preload frequently used datasets onto fast local disks - Use data prefetching techniques (e.g., PyTorch `DataLoader` with `prefetch_factor`) ##### 4. Mounting AI Data in Kubernetes Kubernetes pods can access: - **Persistent Volumes (PVs)** mounted via: - NFS - iSCSI - CSI drivers - **Object storage** (e.g., MinIO, S3) via SDKs or FUSE - Shared network storage (e.g., Lustre via hostPath or CSI plugin) Example (Persistent Volume Claim for NFS): ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ai-data spec: accessModes: - ReadWriteMany resources: requests: storage: 500Gi ``` #### Deployment Automation Tools Manual deployment is error-prone. In real AI environments, use **automation tools** to ensure: - Consistency across clusters - Reproducibility - Scalability ##### 1. Terraform - **Infrastructure-as-Code (IaC)** for provisioning: - VMs - Storage - Networking - Common for **cloud-based AI infrastructure** (AWS, Azure, GCP) Example: ```hcl resource "aws_instance" "gpu_worker" { ami = "ami-12345678" instance_type = "p4d.24xlarge" } ``` ##### 2. Ansible - Automates software setup: - GPU driver installation - Docker/NVIDIA runtime setup - BCM agent installation - Slurm configuration Example playbook: ```yaml - name: Install NVIDIA driver apt: name: nvidia-driver-535 state: present ``` ##### 3. Helm - Kubernetes package manager - Use for: - Deploying **GPU Operator** - Installing **monitoring stacks** (Prometheus, Grafana) - Managing custom apps with config values Example: ```bash helm install gpu-operator nvidia/gpu-operator -n gpu-system ``` ##### 4. GitOps with ArgoCD or Flux - Store **Kubernetes manifests in Git** - Automatically sync deployments on change - Enables **version control** for infrastructure Use this for: - **Production-grade MLOps** - **Secure deployment pipelines** #### Summary of Skills You Must Master | Task | Tools/Skills Required | | ------------------------------- | -------------------------------------------------- | | Set up fast AI storage | Lustre, BeeGFS, NFS, GPFS, local NVMe | | Optimize data access | Use prefetching, RDMA, local caching | | Automate GPU node deployment | Ansible, Terraform | | Automate Kubernetes apps | Helm, GitOps (ArgoCD/Flux) | | Integrate with storage in K8s | PVCs, CSI drivers, object store SDKs | | Validate end-to-end performance | Monitor training speed, GPU utilization, I/O stats | #### Installation & Deployment – Final Review Table | Subtopic | Key Tools / Commands | | ------------------------ | ------------------------------------------------ | | GPU Driver Installation | `nvidia-smi`, `apt`, CUDA toolkit | | Container Runtime | Docker, NVIDIA Container Toolkit | | Kubernetes GPU Setup | GPU Operator, `kubectl`, `nvidia.com/gpu` | | BCM Cluster Management | `bcminit`, Slurm, BCM Web UI | | DOCA & SmartNICs | BlueField, DOCA SDK, Secure Edge Inference | | Fleet Command Deployment | Secure tokens, remote deployment, telemetry logs | | Magnum IO | NCCL, GPUDirect RDMA, multi-node communication | | Storage for AI | Lustre, RDMA, PVCs, NVMe | | Deployment Automation | Terraform, Ansible, Helm, GitOps | --- # Troubleshooting and Optimization ### 1. GPU Monitoring and Diagnostics #### Why This Matters In high-performance AI environments, GPUs are your **most valuable and expensive resources**. You must ensure they are: - **Healthy** - **Efficiently utilized** - **Properly monitored** Misconfiguration, overheating, ECC errors, or memory issues can silently reduce performance or cause job failures. #### Tool 1: DCGM (Data Center GPU Manager) DCGM is a **low-overhead GPU telemetry and diagnostics framework** from NVIDIA. It can be run: - Directly on **bare-metal** - Inside **Docker containers** - In **Kubernetes environments** as part of GPU Operator stack ##### Use Cases of DCGM: | Use Case | Example | | ------------------------------- | ------------------------------------------- | | Monitor GPU temperature | Alert if above 85°C | | Track ECC errors | Detect GPU memory instability | | Enforce power or thermal limits | Trigger alerts on breach | | Monitor application behavior | Collect metrics like memory, SM, PCIe usage | | Export metrics to Prometheus | Build dashboards with Grafana | ##### Key DCGM Commands 1. **List Available GPUs**: ```bash dcgmi discovery -l ``` 2. **Run Health Checks**: ```bash dcgmi health -c ``` This checks: - Temperature - PCIe connectivity - Power - ECC 3. **Get Performance Stats**: ```bash dcgmi stats --groupId 0 ``` This gives: - GPU utilization - Memory usage - PCIe throughput - SM clock rate - Power draw 4. **Collect I/O and NVLink Counters**: DCGM supports advanced custom metrics: - **DCGM_FI_DEV_NVLINK_THROUGHPUT_TX** - **DCGM_FI_DEV_POWER_USAGE** These can be scraped into **Prometheus exporters**. #### Tool 2: `nvidia-smi` – Your GPU Swiss Army Knife A command-line tool installed with the NVIDIA driver. ##### Basic Usage: ```bash nvidia-smi ``` This displays: - Driver version - CUDA version - GPU name - Memory usage - Power draw - Temperature - Active processes ##### Advanced Commands 1. **View Topology and NVLink Connectivity**: ```bash nvidia-smi topo -m ``` Outputs a matrix showing GPU interconnect speeds and NVLink links: ``` GPU0 GPU1 CPU Affinity GPU0 X NV# 0-19 GPU1 NV# X 0-19 ``` 2. **Continuous Monitoring (like `top` for GPUs)**: ```bash nvidia-smi dmon ``` Shows live output of: - GPU utilization - Memory usage - Encoder/decoder usage - PCIe stats ##### Check MIG Configurations: ```bash nvidia-smi mig -lgi ``` List GPU instances created via MIG. #### Example Scenario **Symptom**: GPU memory is full, job crashes. **Diagnosis**: 1. Use `nvidia-smi` to check memory usage in real-time. 2. Use `dcgmi stats` to observe if memory pressure is rising over time. 3. If running in Kubernetes: export metrics via **DCGM exporter** and observe in **Grafana**. **Action**: - Reduce batch size - Switch to mixed precision (FP16) - Fix data loader (e.g., infinite loop) #### Summary of Skills for GPU Monitoring | Tool | Purpose | | ------------------ | -------------------------------------- | | `nvidia-smi` | General GPU monitoring and topology | | `nvidia-smi dmon` | Real-time stats like GPU load & memory | | DCGM CLI (`dcgmi`) | Deeper health metrics and diagnostics | | Prometheus + DCGM | Dashboards for cluster-wide monitoring | ### 2. Docker & Container Issues #### Why Docker Integration Is Critical Most AI workloads today run in **containers** to ensure: - Portability - Dependency isolation - Easy deployment However, if Docker is not properly configured to **interface with NVIDIA GPUs**, containers may: - Fail to detect GPUs - Crash during runtime - Perform poorly (e.g., no GPU acceleration) #### Common Docker GPU Failures | Symptom | Possible Cause | | ------------------------------------- | -------------------------------------- | | `CUDA device not found` | NVIDIA runtime not enabled | | Container runs, but uses only CPU | Started without `--gpus all` flag | | `nvidia-smi` fails in container | Missing drivers or runtime | | MIG mode GPU not visible in container | MIG instance not assigned to container | #### Fix 1: Always Start Containers With GPU Access When running containers, **explicitly request GPUs**: ```bash docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi ``` You can limit to 1 GPU with `--gpus device=0` Or assign a MIG instance directly using `--gpus '"device=UUID"'` #### Fix 2: Validate NVIDIA Container Runtime Ensure Docker is configured to use `nvidia` runtime. ##### Check available runtimes: ```bash docker info | grep -i runtime ``` You should see: ``` Runtimes: runc nvidia Default Runtime: runc ``` If `nvidia` is missing, install NVIDIA Container Toolkit. ##### To test runtime: ```bash docker run --rm --runtime=nvidia nvidia/cuda:12.0-base nvidia-smi ``` If that works, the runtime is installed correctly. #### Fix 3: Check NVIDIA Container Toolkit Installation ##### Steps for Ubuntu: ```bash sudo apt update sudo apt install -y nvidia-container-toolkit sudo systemctl restart docker ``` Also verify: - Kernel modules loaded: `nvidia`, `nvidia_uvm`, `nvidia_drm` - Docker version ≥ 19.03 #### Fix 4: Debugging MIG Containers ##### Problem: MIG instance not accessible in container Fix: - Make sure MIG instance is created using `nvidia-smi mig -cgi` - Assign instance to container with exact **GPU UUID**: ```bash docker run --gpus '"device=GPU-UUID"' ... ``` Use this command to list instances: ```bash nvidia-smi -L ``` #### Real-World Debugging Example **Symptom**: A container launches but shows no GPU available ##### Step-by-step: 1. Inside container: ```bash nvidia-smi ``` → Fails 2. On host: ```bash nvidia-smi # works docker info # check runtime ``` 3. Fix: ```bash sudo apt install nvidia-container-toolkit sudo systemctl restart docker ``` Re-run: ```bash docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi ``` #### Summary of Skills for Container Troubleshooting | Skill | Tool / Command | | ------------------------------- | ---------------------------------------------- | | Confirm GPU access in container | `nvidia-smi` inside container | | Use GPU-enabled container | `docker run --gpus all ...` | | Check runtime support | `docker info | | Assign MIG GPU instance | Use `--gpus "device=UUID"` and `nvidia-smi -L` | | Install NVIDIA runtime properly | `nvidia-container-toolkit`, restart Docker | ### 3. Kubernetes Troubleshooting #### Why GPU Troubleshooting in Kubernetes Is Tricky Kubernetes abstracts away hardware details. So when something goes wrong, such as: - Pods not seeing GPUs - Jobs stuck in `Pending` - GPU metrics not being collected You’ll need to **dig into multiple layers**: - Node configuration - Pod YAML - NVIDIA plugins - Cluster resource scheduling #### Common GPU-Related Kubernetes Issues | Symptom | Likely Root Cause | | --------------------------------- | -------------------------------------------------------- | | Pod does not detect GPU | `nvidia-device-plugin` not running or misconfigured | | Pod stuck in `Pending` | No node with available GPU or incorrect `resources` spec | | Device plugin DaemonSet failed | NodeSelector or taint mismatch, missing runtime support | | Metrics not showing in Prometheus | `dcgm-exporter` not running or not scraped properly | #### Fix 1: Check if GPU Plugin Is Running NVIDIA Device Plugin is deployed as a **DaemonSet**. Check its status: ```bash kubectl get daemonset -n gpu-operator ``` You should see: ``` nvidia-device-plugin-daemonset DESIRED: X READY: X ``` Check logs: ```bash kubectl logs -n gpu-operator ``` If it fails to start: - Check `nvidia-smi` on the node - Check Docker runtime (`nvidia` must be available) #### Fix 2: Verify Pod Resource Requests Kubernetes **won’t schedule** a GPU pod unless you request the GPU correctly: ##### Example Pod YAML: ```yaml resources: limits: nvidia.com/gpu: 1 ``` Missing this will result in the pod being scheduled **without GPU access**. Also confirm: - The node has available GPUs - You are not exceeding resource quotas #### Fix 3: Check Node Labels and Taints Device Plugin only works on nodes that: - **Have GPU hardware** - Are **correctly labeled** (`nvidia.com/gpu.present=true`) - Do **not have taints** that block the DaemonSet (or have matching tolerations) ##### Check node label: ```bash kubectl get nodes --show-labels ``` ##### Check taints: ```bash kubectl describe node | grep Taint ``` If there's a taint like: ``` nvidia.com/gpu=present:NoSchedule ``` Make sure your pod has: ```yaml tolerations: - key: "nvidia.com/gpu" operator: "Exists" effect: "NoSchedule" ``` #### Fix 4: Troubleshoot with `kubectl describe` and Logs If a pod fails to start: ```bash kubectl describe pod ``` Look for: - Failed scheduling - Events at the bottom - Misconfigured requests To inspect logs: ```bash kubectl logs ``` To debug the container runtime interface: ```bash crictl ps -a ``` #### Real-World Debugging Scenario **Symptom**: Pod is in `Pending` state with GPU request. **Diagnosis**: ```bash kubectl describe pod train-gpu-job ``` Output: ``` 0/4 nodes are available: 4 Insufficient nvidia.com/gpu. ``` → No node has free GPU **Fix**: - Free GPU on a node - Add new GPU-enabled node - Reduce `nvidia.com/gpu` request #### Summary of Kubernetes Troubleshooting Skills | Task | Command / Method | | ------------------------------------ | ------------------------------------------------ | | Check GPU DaemonSet | `kubectl get daemonset -n gpu-operator` | | Inspect GPU plugin logs | `kubectl logs` on plugin pod | | Validate resource requests | Pod YAML must include `limits.nvidia.com/gpu: 1` | | Check node labels and taints | `kubectl describe node` | | Troubleshoot pod scheduling failures | `kubectl describe pod`, review Events section | | Confirm runtime is GPU-compatible | `docker info`, `nvidia-smi` on the node | ### 4. Performance Bottlenecks #### Overview Even when infrastructure is working correctly, AI workloads can suffer from: - **Low GPU utilization** - **High job runtime** - **Unexpected memory errors** - **Slow I/O throughput** To fix this, you must **profile all layers** of the stack: - CPU - Memory - GPU - Storage - Interconnect #### 1. CPU Bottlenecks ##### Symptoms: - GPU is idle or lightly loaded - Data preprocessing is slow - CPU usage near 100% on one or few threads ##### Tools: - `htop`: View CPU cores, load balance - `top`: General process-level view - `mpstat -P ALL`: Per-core CPU usage - **Prometheus + Grafana**: Cluster-wide CPU visualization ##### Fixes: - Increase data loading workers (e.g., `num_workers` in PyTorch) - Move preprocessing to GPU (e.g., use NVIDIA DALI) - Balance threads across cores #### 2. Memory Bottlenecks ##### Symptoms: - Pod or container killed with `OOMKilled` - Frequent swapping or paging - `dmesg` shows out-of-memory (OOM) killer activity ##### Diagnostic Commands: ```bash dmesg | grep -i oom free -h vmstat 1 ``` ##### Fixes: - Increase memory allocation in Kubernetes: ```yaml resources: requests: memory: "16Gi" ``` - Optimize memory usage in code - Cache data properly instead of reading repeatedly #### 3. GPU Bottlenecks ##### Symptoms: - GPU is underutilized (< 30%) despite active training - Training job is slow or stuck on data loading ##### Tools: - `nvidia-smi dmon`: View real-time GPU stats - `dcgmi stats`: Monitor streaming multiprocessor (SM) activity - **Nsight Systems**: Timeline of kernel execution and memory copies ##### Common Causes: - Small batch sizes - Data loader too slow - Poor kernel performance (e.g., low occupancy) ##### Fixes: - Use **larger batches** (if memory allows) - Use **mixed precision (FP16)** for better throughput - Optimize data pipeline - Use faster augmentation tools (e.g., DALI) #### 4. I/O Bottlenecks ##### Symptoms: - GPU spends time waiting for data - Job is I/O-bound, not compute-bound - Logs show data not loading in time ##### Diagnostic Tools: - `iostat`, `iotop`: Disk read/write stats - `nvidia-smi dmon`: Look for spikes in memory throughput - **DCGM custom metrics** for I/O counters - Nsight Systems: Shows memory copy stalls ##### Fixes: - Use **parallel file systems** (Lustre, BeeGFS) - Enable **RDMA** if supported - Prefetch data in code - Use **TFRecords**, **LMDB**, or **WebDataset** for optimized storage formats #### Real-World Scenario **Symptom**: GPU utilization is only 20% during training **Diagnosis**: 1. `nvidia-smi dmon` → GPU not active 2. `top` → Python process active, CPU at 100% 3. Python log shows: “Loading batch…” **Cause**: Data loading is the bottleneck **Fix**: - Increase `num_workers` in `DataLoader` - Move preprocessing to GPU with NVIDIA DALI #### Summary: Common Bottlenecks & Fixes | Bottleneck Type | Symptom | Tool(s) | Common Fixes | | --------------- | -------------------------------- | ------------------------------ | --------------------------------------------------- | | CPU | High wait time, low GPU usage | `htop`, `mpstat`, Prometheus | Tune data loader, use GPU-accelerated preprocessing | | Memory | OOMKilled, swap, slow system | `dmesg`, `free`, `vmstat` | Allocate more memory, optimize memory usage | | GPU | Low utilization, slow training | `nvidia-smi`, `Nsight Systems` | Increase batch size, use FP16, optimize kernels | | I/O | Data stalls, slow dataset access | `iostat`, `Nsight`, DCGM | Use RDMA, faster storage, caching, better formats | ### 5. Interconnect Analysis (NVLink & NVSwitch) #### Why GPU Interconnects Matter In **multi-GPU** systems (like DGX/HGX servers), GPUs need to **exchange data at high speeds**—especially during: - Distributed training (e.g., gradient sharing) - Model parallelism (splitting parts of models across GPUs) - Tensor or batch parallel computation Slow or misconfigured interconnects lead to: - **High latency** - **Communication bottlenecks** - **Uneven performance across GPUs** #### NVLink: High-Speed GPU-to-GPU Communication ##### What is NVLink? - A **direct, high-bandwidth connection** between NVIDIA GPUs - Faster than PCIe (up to 600 GB/s bidirectional on newer models) - Used in most modern **HGX** and **DGX** systems ##### Key Features: | Feature | Description | | ------------------- | ----------------------------------------------- | | Peer-to-peer access | GPUs can directly access each other’s memory | | Transparent to user | No code changes needed if using NCCL or CUDA | | Scalable | Supports topologies up to 8 GPUs interconnected | ##### View NVLink Topology Use: ```bash nvidia-smi topo -m ``` Example output: ``` GPU0 GPU1 GPU2 GPU0 X NV1 NV2 GPU1 NV1 X SYS GPU2 NV2 SYS X ``` Legend: - `NV1`, `NV2`: Connected via NVLink - `SYS`: Communication via PCIe (slower) ##### Interpret This Output: - **Direct NVLink** between GPUs is best - If traffic goes over **SYS**, performance will degrade #### NVSwitch: Full-Bandwidth All-to-All Fabric ##### What is NVSwitch? - A **switching fabric** that allows **any GPU to talk to any other GPU** at full NVLink speed - Used in systems like **DGX A100**, **DGX H100**, or **HGX-8/16** - Replaces the limited point-to-point NVLink architecture ##### Why it Matters: - Uniform communication bandwidth - Reduces “topology bottlenecks” - No need to worry about which GPU talks to which #### NCCL Tests: Measure Interconnect Performance NCCL (NVIDIA Collective Communication Library) is optimized for: - **AllReduce**, **Broadcast**, **Gather**, **Scatter** - Multi-GPU training frameworks like PyTorch, TensorFlow, Horovod ##### To Run Bandwidth/Latency Tests: Clone and build: ```bash git clone https://github.com/NVIDIA/nccl-tests.git cd nccl-tests make MPI=1 ``` Run a simple test: ```bash mpirun -np 4 ./build/all_reduce_perf -b 8 -e 512M -f 2 -g 1 ``` This will show: - Bandwidth (GB/s) - Latency per GPU pair ##### Tip: Run tests between GPUs with and without NVLink to **compare performance**. #### Real-World Debugging Example **Symptom**: Multi-GPU training is slower on some nodes. ##### Steps: 1. Run `nvidia-smi topo -m` → GPU2 communicates over `SYS`, others via `NVLink` 2. Run NCCL tests: → Lower bandwidth between GPU2 and others 3. Fix: - Rewire NVLink bridge (if physical) - Reassign workload to NVLink-connected GPUs only #### Summary: Interconnect Troubleshooting | Tool / Method | Use Case | | ---------------------- | ----------------------------------------- | | `nvidia-smi topo -m` | View GPU connectivity and NVLink layout | | NCCL Tests | Benchmark GPU-to-GPU bandwidth | | Nsight Systems | Identify communication stalls | | DCGM Bandwidth Metrics | Export NVLink/PCIe stats over time | | Hardware Check | Confirm NVLink bridges or NVSwitch status | #### Best Practices | Best Practice | Why It Matters | | -------------------------------------------- | ------------------------------------------------ | | Place jobs on NVLink-connected GPUs | Ensures fast peer-to-peer communication | | Monitor GPU interconnect bandwidth | Avoid silent slowdowns during training | | Use NVSwitch if available | Guarantees full-bandwidth across all GPUs | | Run NCCL benchmarks after cluster deployment | Validate performance before real training begins | ### 6. Nsight Systems & Nsight Compute #### Why Use Profiling Tools? Even if your code runs and uses GPUs, **performance may still be poor** due to: - Inefficient kernel launches - Poor memory access patterns - Synchronization overhead - I/O stalls or CPU bottlenecks To identify these issues, NVIDIA offers two powerful tools: | Tool | Focus Level | What It Analyzes | | ------------------ | ------------------- | ---------------------------------------------- | | **Nsight Systems** | System-wide | Timeline of CPU, GPU, memory, I/O, and kernels | | **Nsight Compute** | Kernel-level (CUDA) | Thread occupancy, memory coalescing, warps | #### Nsight Systems ##### Purpose: - Provides a **global view** of your application - Helps spot issues across CPU, GPU, and system resources ##### Key Features: - Timeline view of: - Kernel launches - Memory copies - CPU thread activity - API calls (e.g., cuMemcpy, cudaLaunchKernel) - Zoom into specific regions to spot stalls ##### Launch via CLI: ```bash nsys profile -o output_name ./your_script.sh ``` Or for Python: ```bash nsys profile -o tf_model python train.py ``` ##### Output: Generates `.qdrep` and `.nsys-rep` files, viewable in: - **Nsight Systems GUI** - Or CLI summary: ```bash nsys stats output_name.qdrep ``` ##### Common Issues Nsight Systems Reveals: | Symptom | Interpretation | | ----------------------------- | ------------------------------------------ | | Kernel launches delayed | CPU bottleneck or sync blocking | | Long memory copies | Poor data locality or missing page pinning | | GPU idle while CPU active | Data not arriving fast enough | | Lots of small kernel launches | Launch overheads outweigh compute gains | #### Nsight Compute ##### Purpose: - Dive into **individual kernel performance** - Understand what limits your GPU execution ##### Key Metrics: | Metric | What It Tells You | | ----------------------------- | ----------------------------------------- | | **Occupancy** | GPU thread block utilization | | **Memory Coalescing** | Whether threads access memory efficiently | | **Warp Execution Efficiency** | Fraction of warps that are active | | **Shared Memory Utilization** | Helps tune for memory-bound kernels | ##### Launch via CLI: ```bash nsys profile -t cuda ./app ``` To analyze a single kernel: ```bash nv-nsight-cu-cli ./app ``` Or open the `.ncu-rep` in **Nsight Compute GUI** for visual profiling. ##### Common Issues Nsight Compute Detects: | Issue | Description | Suggested Fix | | ---------------------- | ----------------------------------- | ------------------------------------------ | | Low occupancy | Blocks too small, too few threads | Increase thread count or adjust block size | | Poor memory coalescing | Unaligned memory access | Use contiguous memory layouts | | Branch divergence | Threads follow different code paths | Refactor kernel logic | | High register pressure | Too many registers per thread | Optimize variable usage or split kernels | #### Example Use Case: PyTorch Model is Slow 1. Run: ```bash nsys profile -o myrun python train.py ``` 2. Inspect Timeline: - GPU is idle most of the time - `cudaMemcpyAsync` takes 80% of time → Suggests data is not on GPU or input pipeline is too slow 3. Fix: - Move preprocessing to GPU (e.g., DALI) - Use pinned memory for data transfers - Batch memory copies #### Summary: When to Use Each Tool | Tool | Use Case | | -------------- | ------------------------------------------------- | | Nsight Systems | End-to-end profiling, timeline view | | Nsight Compute | In-depth kernel analysis, tuning CUDA performance | #### Practical Tips | Tip | Why It Helps | | ----------------------------------------------- | ---------------------------------------------- | | Always profile after major code changes | Prevent hidden regressions | | Use Nsight in early development, not just final | Catch architectural mistakes early | | Pair with Prometheus + Grafana for correlation | Relate kernel issues with system-level metrics | | Use multiple runs to find consistent patterns | Avoid noise from one-off spikes or jitter | ### 7. Training Optimization Strategies #### Why Optimization Matters Training large AI models is **expensive and time-consuming**. Without optimization: - Training takes days instead of hours - GPU resources are underutilized - Scaling across GPUs becomes inefficient By optimizing training, you can: - **Reduce training time** - **Lower compute cost** - **Improve model convergence stability** #### 1. Data Pipeline Optimization ##### Symptoms of Poor Data Pipeline: - GPU idle time is high - Training slow despite low CPU load - I/O waits visible in profiling tools (e.g., Nsight Systems) ##### Optimization Techniques: | Technique | Benefit | | ------------------------- | ---------------------------------------------- | | **Data prefetching** | Fetch next batch while current is training | | **Parallel data loading** | Use `num_workers` > 0 in PyTorch | | **Efficient formats** | Use **TFRecords**, **WebDataset**, or **LMDB** | | **GPU-based decoding** | Use **NVIDIA DALI** for image/video data | | **Data sharding** | Split data across nodes to reduce I/O | ##### PyTorch Example: ```python DataLoader(dataset, batch_size=32, num_workers=8, prefetch_factor=2) ``` #### 2. I/O Optimization Slow data loading from storage (especially NFS) can bottleneck training. ##### Solutions: - Use **parallel file systems** (Lustre, BeeGFS, GPFS) - Use **local NVMe SSDs** for temporary caching - Enable **RDMA** for remote file access - Use **GPUDirect Storage** if supported #### 3. GPU Usage Optimization Most modern GPUs support **mixed precision** training (e.g., FP16 or BF16), which improves performance. ##### Techniques: | Optimization | Benefit | | ------------------------- | ------------------------------------ | | **Mixed precision** | Use Tensor Cores, reduce memory | | **Larger batch sizes** | Better GPU utilization | | **Gradient accumulation** | Simulate large batches on small GPUs | ##### PyTorch Example: ```python from torch.cuda.amp import GradScaler, autocast scaler = GradScaler() for data in loader: with autocast(): output = model(data) loss = loss_fn(output, target) scaler.scale(loss).backward() scaler.step(optimizer) scaler.update() ``` #### 4. Distributed Communication Optimization When using multiple GPUs/nodes, communication can dominate training time. ##### Techniques: | Method | Tool / Library | Benefit | | ------------------------------ | ----------------------------- | ----------------------------- | | **AllReduce optimization** | NCCL | Faster gradient sync | | **Horovod Fusion Buffer** | Horovod | Reduces network overhead | | **Overlapping compute & comm** | Native PyTorch DDP or Horovod | Better GPU utilization | | **Use NVLink/NVSwitch** | Hardware | Low-latency inter-GPU traffic | #### 5. Scheduling Strategy If multiple jobs or users share a cluster, poor scheduling can result in: - Resource starvation - GPU fragmentation - Overhead from context switching ##### Best Practices: | Technique | Benefit | | -------------------------------------- | ------------------------------------ | | **Bind jobs to NVLink-connected GPUs** | Maximizes inter-GPU speed | | **Set job affinity** (K8s, Slurm) | Avoids placement on overloaded nodes | | **Use MIG** (on A100/H100) | Enables fair GPU partitioning | | **Prioritize critical jobs** | Use Slurm QOS or K8s priority class | #### Summary: Training Optimization Techniques | Area | Method | Tool/Tech Used | | ------------- | ------------------------------------- | ------------------------------ | | Data | Prefetch, shard, parallel load | DALI, PyTorch DataLoader | | I/O | Parallel FS, RDMA, caching | BeeGFS, GPUDirect Storage | | GPU | FP16, batch size, accumulation | PyTorch AMP, Tensor Cores | | Communication | AllReduce, NCCL, overlapping transfer | NCCL, Horovod, DDP | | Scheduling | Topology-aware scheduling | NVLink, MIG, Kubernetes, Slurm | ### 8. Logs, Debugging Tools & Common Error Types #### 1. Why Logs Are Essential In production environments, **logs are your first line of defense** when something breaks. They help answer: - Why did my job crash? - Was it an infrastructure issue or a code bug? - Did the system run out of memory, GPU, or file descriptors? #### 2. Key Logging and Debugging Tools | Tool | Use Case | | -------------------- | --------------------------------------------- | | `journalctl` | System-wide logs (including Docker and Slurm) | | `docker logs ` | Container-specific logs | | `kubectl logs ` | Kubernetes container logs | | Slurm logs | Scheduler + compute node logs | | `dmesg` | Kernel-level logs (e.g., OOM errors) | ##### System Logs ```bash journalctl -xe ``` Use for checking driver issues, hardware failures, or runtime crashes. ##### Docker Logs ```bash docker ps -a docker logs ``` Check for: - CUDA errors - Missing libraries - Runtime crashes ##### Kubernetes Logs Check the logs of a pod: ```bash kubectl logs ``` For multi-container pods: ```bash kubectl logs -c ``` Describe pod to see failure reasons: ```bash kubectl describe pod ``` ##### Slurm Logs Slurm logs are usually found in: ```bash /var/log/slurm/slurmctld.log /var/log/slurm/slurmd.log ``` Useful when: - Jobs are not starting - Node goes offline - GPU resources not allocated correctly #### 3. Common Error Types You Should Recognize Here’s a cheat sheet of the most frequent runtime errors you’ll see and how to interpret them. ##### GPU Not Found **Error Message:** ``` CUDA device not available ``` **Diagnosis:** - NVIDIA driver not loaded - Docker container lacks `--gpus` flag - MIG instance not assigned **Fix:** - Run `nvidia-smi` on host and in container - Check runtime settings - Use `--gpus all` or `--gpus "device=UUID"` ##### Out-of-Memory (OOM) **Error Message:** ``` Killed: Out of memory ``` **Diagnosis:** - Container memory request too small - Job exceeded available RAM or GPU memory **Fix:** - Increase `resources.requests.memory` in K8s - Tune batch size or model size - Use mixed precision (FP16) ##### GPU Device Busy **Error Message:** ``` GPU already in use ``` **Diagnosis:** - Another process is using GPU - Container trying to access unavailable MIG partition **Fix:** - Use `nvidia-smi` to inspect usage - Kill unused processes - Reschedule job to another node ##### Job Hangs / No Output **Symptoms:** - Job appears to run but no logs or model output **Diagnosis:** - Inter-GPU communication bottleneck - Infinite loop or deadlock in code - Blocked I/O **Fix:** - Profile with Nsight Systems - Add debug-level logging - Test components in isolation #### Final Summary: Must-Have Debugging Skills | Issue Type | Tool / Method | | --------------- | ------------------------------------------- | | Crash Diagnosis | `journalctl`, `docker logs`, `kubectl logs` | | GPU Access | `nvidia-smi`, container runtime settings | | Memory Issues | `dmesg`, Prometheus memory metrics | | Slurm Failures | `slurmctld.log`, `squeue`, `sinfo` | | Stuck Jobs | Profiling tools + log inspection | --- # Administration (Additional Content) ### 1. What to Expect in the Exam (Question Style Overview) The **"Administration"** domain typically tests foundational GPU system skills using standard exam formats. Here's what you can expect: #### Typical Question Formats | Format Type | Description | Example | | ---------------------- | --------------------------------------------- | ---------------------------------------------- | | Multiple Choice | Select the best command or explanation | "Which command lists all GPUs?" | | Command Identification | Identify what a command does | "What does `dcgmi health -c` do?" | | Scenario-Based | Real-world situation, pick the right response | "A node has ECC errors, what tool do you use?" | | Concept Definition | Explain a service or system behavior | "What is the persistence daemon used for?" | #### Commonly Tested Skills - Recognizing the output and use of `nvidia-smi`, `dcgmi`, `systemctl` - Verifying NVIDIA services like `nvidia-persistenced` - Diagnosing node availability using GPU health/status tools - Navigating GPU environment variables or configuration files ### 2. Command Mnemonics and Memorization Techniques #### Command Mnemonics | Command | Function | Memory Aid | | -------------------------------------- | --------------------------- | ------------------------------------------ | | `dcgmi discovery -l` | Lists all visible GPUs | “D for discovery, L for list” | | `dcgmi health -c` | Health check of all devices | “C = Check” | | `nvidia-smi` | View driver, GPU usage | Think “System Monitoring Interface” | | `nvidia-smi topo -m` | Shows inter-GPU topology | “topo = topology”, output = matrix | | `systemctl status nvidia-persistenced` | Shows daemon status | Always use `systemctl status` for services | #### Flashcard Style Learning Use flashcards with **command on one side**, and **purpose + memory hook** on the other: - **Front:** `dcgmi stats --groupId 0` - **Back:** *Purpose*: Display performance metrics (utilization, PCIe, memory) *Mnemonic*: Stats for Group 0 = Node-wide statistics Tools like Anki make digital flashcards efficient for spaced repetition. #### Categorized Command Sets Organize your study by function to retain better: | Category | Key Commands | | ---------------------- | --------------------------------------- | | GPU Discovery & Health | `dcgmi discovery -l`, `dcgmi health -c` | | Monitoring | `nvidia-smi`, `nvidia-smi dmon` | | Topology | `nvidia-smi topo -m` | | Service Management | `systemctl status nvidia-persistenced` | --- # Workload Management (Additional Content) ### **1. Kubernetes – GPU Workload Management** #### `nvidia-device-plugin` YAML Structure The NVIDIA device plugin is typically deployed as a **DaemonSet**. Key YAML components include: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: nvidia-device-plugin-daemonset spec: template: spec: containers: - name: nvidia-device-plugin-ctr image: nvcr.io/nvidia/k8s-device-plugin:latest args: ["--mig-strategy=single"] env: - name: FAIL_ON_INIT_ERROR value: "false" volumeMounts: - name: device-plugin mountPath: /var/lib/kubelet/device-plugins ``` **Key fields:** | Field | Purpose | | -------------------- | -------------------------------------------------- | | `--mig-strategy` | Controls MIG handling (`none`, `single`, `mixed`) | | `FAIL_ON_INIT_ERROR` | Allows graceful start if no GPU exists | | `volumeMounts` | Mount path for Kubernetes to access plugin sockets | #### Multi-Container Pod GPU Behavior In Kubernetes, **GPU resources are allocated at the Pod level**: - All containers in the same Pod **share** the same GPU allocation. - There is **no fine-grained control** to split 1 GPU across containers in the same Pod (unless using MIG). - Default behavior: GPU is made visible to **all containers** via environment variable `NVIDIA_VISIBLE_DEVICES`. #### Jobs vs. Pods | Type | Behavior | | ------- | --------------------------------------------------------------------------- | | **Pod** | One-off unit, no restart on failure | | **Job** | Handles retries, status tracking, suitable for training or evaluation tasks | **Best practice:** Use `Job` for batch inference or training workloads; only use `Pod` for debugging or non-repeatable jobs. #### Fractional GPU / MIG Support in Kubernetes - MIG support is available via the **NVIDIA device plugin** with `--mig-strategy`. - Time-slicing (e.g., A10 GPU) is **experimental** with newer Kubernetes + NVIDIA driver versions. - Still **no native fractional GPU (e.g., 0.5 GPU)** support in Kubernetes without MIG. #### GPU Request and Limit Behavior | Field | Effect | | ----------------------------------- | -------------------------------- | | `resources.requests.nvidia.com/gpu` | Used for **scheduling** | | `resources.limits.nvidia.com/gpu` | Used for **runtime enforcement** | **Rules:** - They **must match** exactly (`1` and `1`), or **scheduling fails**. - No overcommit allowed on GPU. - Omitting `limits` but setting `requests` → OK (GPU allocated, but no enforced limit). - Setting `limits > requests` → Invalid. ### **2. Slurm Workload Management** #### Job Arrays in Slurm Used to submit **multiple similar jobs** with a single `sbatch` command: ```bash sbatch --array=0-9 run_train.sh ``` Inside the script, access the job index with `$SLURM_ARRAY_TASK_ID`. **Use cases:** Hyperparameter sweep, different seeds, model variant evaluation. #### `srun` vs `sbatch` | Command | Usage | | -------- | -------------------------------------------------------- | | `sbatch` | Submit **batch** job, executes in background | | `srun` | Run **interactively** or within sbatch for MPI/multinode | **Interactive example:** ```bash srun --gres=gpu:1 --pty bash ``` #### `scontrol` for Job Management Use `scontrol` to modify queued or running jobs: - Suspend: `scontrol suspend ` - Resume: `scontrol resume ` - Change time limit: `scontrol update jobid= TimeLimit=01:00:00` #### Partition Strategy Optimization | Partition Attribute | Behavior | | ------------------- | ---------------------------------------- | | `Priority` | Determines job execution order | | `PreemptMode` | Allows interruption of low-priority jobs | | `TimeLimit` | Controls job length per partition | **Best practice:** - Use **short partitions** for testing - Use **preemption** to enable high-priority job insertion #### DCGM + Prometheus in Slurm - Deploy **dcgm-exporter** on each node. - Collect metrics via **Prometheus scrape**. - Map GPU usage to job/user via **Slurm accounting + hostname mapping**. ### **3. Multi-Tenancy and GPU Isolation** #### Slurm + MIG Binding You can assign **specific MIG slices** to users via `GRES` and `Job Constraints`: ```bash #gres.conf Name=gpu Type=MIG-GPU-0 UUID=GPU-abc123 Count=1 #slurm.conf NodeName=nodex Gres=gpu:MIG-GPU-0:1 #job script #SBATCH --gres=gpu:MIG-GPU-0:1 ``` #### GPU Usage Auditing Use **SlurmDBD** to collect per-user job GPU metrics: ```bash sacct -u --format=JobID,Elapsed,ReqGRES,AllocGRES,MaxRSS ``` Integrate with **Grafana**, **Elastic**, or **custom scripts** for department-level billing. #### Kubernetes `LimitRange` for GPU Control Use `LimitRange` to control **min/max GPU per Pod**: ```yaml apiVersion: v1 kind: LimitRange spec: limits: - type: Container max: nvidia.com/gpu: 2 min: nvidia.com/gpu: 1 ``` Pair with `ResourceQuota` to limit total GPU per Namespace. ### **4. Monitoring & Logging** #### DCGM Exporter Metrics (Prometheus) | Metric | Meaning | | ---------------------- | ---------------------------------------------- | | `dcgm_gpu_power_usage` | Instant power draw in watts | | `dcgm_memory_temp` | Memory temperature (°C) | | `dcgm_ecc_errors` | Accumulated ECC errors (corrected/uncorrected) | #### Centralized Pod Log Collection Use tools like: - **Fluentd**, **Fluent Bit** → forward logs from container stdout - **Logstash** → log parsing - **ELK/EFK** stack → storage and visualization Deploy as DaemonSet + ConfigMap. #### Node Problem Detector Integration Add **hardware monitoring hooks** via `node-problem-detector`: - Source: `/var/log/syslog`, `dmesg` - Reports GPU errors, thermal events as **Kubernetes Node Conditions** #### Alertmanager GPU Alerts Sample config: ```yaml groups: - name: GPU overheating rules: - alert: GPUTemperatureTooHigh expr: dcgm_temperature_gpu > 85 for: 2m labels: severity: critical annotations: summary: "GPU Temperature Too High" description: "Node {{ $labels.instance }} GPU exceeds 85°C" ``` ### **5. Resource Quota and Scheduling** #### `LimitRange` vs `ResourceQuota` | Resource | Scope | Function | | --------------- | ----------------- | ---------------------------------------- | | `LimitRange` | Per Pod/Container | Set min/max per resource type | | `ResourceQuota` | Per Namespace | Set total aggregate limit (e.g., 8 GPUs) | Use both to enforce **fair resource usage**. #### Namespace Hierarchy and Subquotas Kubernetes supports **v1beta1 hierarchical quotas** (via plugins like HNC): - Allow subteam quotas: `teamA-dev`, `teamA-prod` - Schedulers can allocate based on **sub-quota priorities** #### Scheduler Extenders Advanced feature to inject **custom scheduling logic** (e.g., for GPU memory size or NUMA topology): - Write HTTP extender server - Modify kube-scheduler config to include: ```yaml extenders: - urlPrefix: "http://localhost:12300/scheduler" filterVerb: "filter" prioritizeVerb: "prioritize" ``` Used by platforms like Volcano or Alibaba AI scheduler. ### **6. Workflow Automation Tools** #### Argo Workflows with GPU Example: ```yaml containers: - name: train image: pytorch/pytorch:2.0 resources: limits: nvidia.com/gpu: 1 ``` Argo will schedule the step only on GPU-capable nodes. #### Katib for Hyperparameter Tuning Example config for tuning learning rate: ```yaml objective: type: maximize goal: 0.9 objectiveMetricName: accuracy parameters: - name: learning_rate parameterType: double feasibleSpace: min: "0.01" max: "0.1" ``` Runs trials using Kubernetes jobs with GPUs. #### Airflow `KubernetesPodOperator` DAG snippet: ```python KubernetesPodOperator( namespace='ai-pipeline', image='my-model:latest', cmds=["python", "train.py"], name="train-task", task_id="train-model", resources={"limit_memory": "4Gi", "limit_gpu": "1"} ) ``` Airflow dynamically creates GPU-enabled Pods. #### Retries & Scheduling In Argo/Kubeflow: ```yaml retryStrategy: limit: 3 retryPolicy: "Always" ``` Supports backoff, retry delay, onFailure handlers for **production reliability**. --- # Installation and Deployment (Additional Content) ### **1. Network Configuration & Debugging in Containers and Kubernetes** #### Purpose: To ensure GPU containers or Pods can access external networks, internal services, and cluster components correctly. #### Docker Network Modes Comparison | Mode | Description | Use Case | | ---------- | ------------------------------------------------ | ----------------------------------------- | | **bridge** | Default mode, NAT-based network with separate IP | Good for isolation; requires port mapping | | **host** | Shares host’s network namespace | Better performance; limited isolation | In `bridge` mode, the container gets a virtual IP and communicates via the host. In `host` mode, the container sees the same IP as the host, which is suitable for high-performance GPU networking (e.g., RDMA). #### Kubernetes Network Plugins Overview | Plugin | Feature Highlights | | ----------- | ------------------------------------------------ | | **Flannel** | Simple, stable; uses VXLAN or host-gw | | **Calico** | Supports network policy, IP-in-IP, BGP | | **Cilium** | eBPF-based, supports L7 security & observability | **Calico** is the most widely used in GPU clusters due to its rich policy support and performance. #### Useful Debugging Commands ```bash ifconfig # Check container or node network interface ip a # View all interface and IP assignments kubectl get pods -o wide # Show Pod IPs and node assignments kubectl exec pod -- ping google.com # Verify external access ``` ### **2. GPU Quotas and Resource Limits in Kubernetes** #### Purpose: To prevent over-allocation and support fair GPU sharing in multi-user environments. #### ResourceQuota Example ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota namespace: ai-team spec: hard: nvidia.com/gpu: "4" ``` This restricts the entire namespace to use no more than **4 GPUs** total. #### LimitRange Example ```yaml apiVersion: v1 kind: LimitRange metadata: name: gpu-defaults namespace: ai-team spec: limits: - type: Container default: nvidia.com/gpu: 1 max: nvidia.com/gpu: 2 ``` This ensures each container in the namespace will use **1 GPU by default**, with a **maximum of 2**. #### Multi-Tenant Isolation Strategy - Use separate **Namespaces** for each team or user group. - Combine with **RBAC policies** to restrict access. - Enforce quotas with `LimitRange + ResourceQuota`. ### **3. Cluster Auto-Validation Script** #### Purpose: Quickly verify that a fresh cluster deployment is healthy and ready for GPU workloads. #### Recommended Bash Script Structure: ```bash #!/bin/bash echo "== Checking GPU ==" nvidia-smi || echo "GPU driver not working" echo "== Checking Docker & GPU Toolkit ==" docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi || echo "GPU container failed" echo "== Checking Kubernetes ==" kubectl get nodes kubectl describe node $(hostname) echo "== Checking BCM (if installed) ==" bcmsystem status || echo "BCM not installed or not initialized" ``` This script validates **GPU drivers**, **container runtime**, **Kubernetes node readiness**, and **optional BCM agent status**. ### **4. Log Debugging & Common Error Diagnosis** | Issue | Root Cause | Solution | | ----------------------------------- | ---------------------------------------------------- | -------------------------------------------------- | | `nvidia-smi` shows no GPU | Driver not installed or kernel mismatch | Run `dmesg`, check `lsmod | | `nvidia-smi` fails inside container | Missing NVIDIA Container Toolkit or `--gpus` not set | Install toolkit, start container with `--gpus all` | | Pod stuck in `Pending` | No GPU node or device plugin not running | Check `kubectl get pods -n gpu-operator` | | BCM node not recognized | Wrong IP or missing join command | Check network, rerun `bcminit --join` | **Diagnostic Tools:** - `journalctl -u kubelet` - `docker logs ` - `kubectl describe pod ` - `kubectl get events` ### **5. Docker Image Optimization Techniques** #### Purpose: Build efficient and portable GPU images for training or inference. #### Sample Dockerfile with CUDA Support ```Dockerfile FROM nvidia/cuda:12.2.0-cudnn8-runtime-ubuntu22.04 RUN apt update && apt install -y python3 python3-pip COPY requirements.txt . RUN pip install -r requirements.txt COPY . /app WORKDIR /app CMD ["python3", "main.py"] ``` #### Optimization Strategies | Technique | Description | | ----------------------------- | ------------------------------------------------ | | **Clean intermediate layers** | `apt clean && rm -rf /var/lib/apt/lists/*` | | **Multi-stage builds** | Compile in one stage, copy only runtime to final | | **Use `docker buildx`** | Enables cross-platform builds (e.g., ARM64) | Example: ```bash docker buildx build --platform=linux/amd64,linux/arm64 -t myimage:gpu . ``` ### **6. GPU Driver and CUDA Compatibility Reference** | CUDA Version | Required Driver Version | Supported Architectures | | ------------ | ----------------------- | ----------------------- | | CUDA 12.2 | ≥ 535.x | Hopper, Ada, Ampere | | CUDA 11.8 | ≥ 510.x | Ampere, Volta, Turing | | CUDA 10.2 | ≥ 440.x | Volta, Turing | **Best Practices:** - Always match CUDA version with **driver version**. - Use `nvidia-smi` to verify driver/CUDA compatibility. - Avoid mixing **too old** drivers with **new CUDA versions**. --- # Troubleshooting and Optimization (Additional Content) ### **1. GPU Monitoring & DCGM Integration** #### DCGM + Prometheus + Grafana Visualization Architecture To understand how GPU metrics flow from GPU hardware to visualization dashboards: **Data Flow Overview:** ``` [NVIDIA GPU Hardware] ↓ [DCGM Exporter DaemonSet] ↓ [Prometheus Scraper] ↓ [Grafana Dashboard] ``` - **DCGM Exporter** collects metrics from `nvidia-smi` and `dcgmi`. - **Prometheus** scrapes metrics via `dcgm-exporter:9400/metrics`. - **Grafana** queries Prometheus and displays GPU utilization, power, temperature, and ECC stats. #### Helm-Based DCGM Exporter Installation To deploy DCGM Exporter via Helm in a Kubernetes GPU cluster: **Sample `values.yaml`:** ```yaml daemonset: enabled: true serviceMonitor: enabled: true resources: limits: nvidia.com/gpu: 1 podAnnotations: prometheus.io/scrape: "true" prometheus.io/port: "9400" ``` Then run: ```bash helm repo add nvidia https://nvidia.github.io/k8s-dcgm helm install dcgm-exporter nvidia/dcgm-exporter -f values.yaml ``` ### **2. Kubernetes GPU Optimization Enhancements** #### Topology-Aware Scheduling (v1.24+) This feature ensures Pods are scheduled based on: - GPU **NUMA locality** - **NVLink connectivity** - **CPU-GPU binding consistency** **Benefits:** - Reduces cross-NUMA latency - Enhances inter-GPU bandwidth - Improves overall GPU pipeline efficiency **Usage:** - Use `TopologyManager` and `CPUManager` in Kubelet. - Combine with NVIDIA Device Plugin using `topologyPolicy: best-effort`. #### NVIDIA GPU Operator Structure Diagram (Simplified) | Component | Role | | ------------------------- | ------------------------------------ | | `nvidia-device-plugin` | Reports GPU devices to kubelet | | `dcgm-exporter` | Exposes GPU metrics to Prometheus | | `nvidia-driver-daemonset` | Ensures host has correct GPU driver | | `mig-manager` | Manages MIG profiles if enabled | | `validator` | Validates installation and readiness | All components are deployed via a **Helm chart** and run as **DaemonSets** or **Pods** on GPU nodes. ### **3. Training Optimization: Memory and Interconnects** #### Memory Fragmentation in MIG Scenarios **Problem:** Fragmented GPU memory may block container startup or reduce performance. **Solutions:** - Use fixed **batch sizes** to stabilize allocation patterns. - Pre-allocate MIG profiles with known shapes (e.g., 1g.5gb). - Use **Nsight Compute** to analyze memory heatmap and pinpoint fragmentation sources. #### GPUDirect Storage & RDMA - **GDS** enables direct data movement from NVMe SSD to GPU without CPU copy. - **RDMA** enables direct memory access between nodes for high-speed networking. **Validation Tools:** - `fio` + `nvidia-fs` for GDS performance - `ib_write_bw` for RDMA throughput **Command Example:** ```bash fio --name=gds-test --filename=/mnt/data/testfile --ioengine=nvidia-fs --rw=read --size=4G ``` ### **4. Nsight Tools Usage and Profiling Diagnosis** #### Nsight Systems `.qdrep` Report Structure **GUI Layout Overview:** - **Top timeline:** GPU kernels, memory copies - **Middle pane:** CPU threads and sync points - **Bottom:** API trace and durations **Diagnostic Use Cases:** | Symptom | Interpretation | | ---------------------------- | ----------------------------------- | | Long `cudaMemcpyAsync` | Bottlenecked I/O | | Gaps between kernel launches | CPU-side delay or sync issue | | Repetitive short kernels | Poor kernel fusion, launch overhead | #### Nsight Compute Kernel Report Interpretation | Metric | Meaning | Optimization | | --------------------------- | ------------------------- | -------------------------- | | **Occupancy < 40%** | Underutilized GPU | Increase threads per block | | **High warp divergence** | Control flow inefficiency | Restructure kernel logic | | **Shared memory underused** | Memory bandwidth waste | Refactor memory access | ### **5. Multi-Tenant GPU Scheduling Strategy** #### NVLink-Aware Job Placement To avoid performance imbalance in multi-GPU nodes: | User | GPU Allocation | | ---- | ------------------------------------ | | A | GPU0–GPU3 (full NVLink interconnect) | | B | GPU4–GPU7 | **How to Configure:** - Kubernetes: Use `nodeSelector` or custom `scheduler extender`. - Slurm: Set `GRES` + topology-aware constraints. #### MIG-Based Multi-User Resource Planning **Diagram Example (A100 40GB):** | MIG Instance | User Assigned | Profile | | ------------ | ------------- | ------- | | GPU0:0 | User A | 1g.5gb | | GPU0:1 | User B | 2g.10gb | | GPU1:0 | User C | 3g.20gb | **Key Principles:** - Match workloads to MIG profiles - Use `nvidia-smi -L` to map UUIDs - Kubernetes: Assign using `--gpus "device=UUID"` ---

Shopping cart

Subtotal:

Examination Introduction

Learning Method

Study Plan

Skill Practice Questions