A 4-Week Roadmap to Mastering AI Infrastructure and GPU-Based Workload Management

The NVIDIA Certified Professional for AI Operations (NCP-AIO) is a highly practical certification that proves your capability to deploy, manage, monitor, and troubleshoot large-scale AI infrastructure using NVIDIA technologies. As AI workloads grow increasingly complex and compute-intensive, professionals with strong skills in GPU orchestration, workload scaling, and infrastructure optimization are in high demand. This study plan is carefully designed to help you become one of them.

This 4-week learning roadmap was built with three goals in mind: clarity, structure, and retention. Whether you're a beginner just entering the world of AI infrastructure or a practitioner aiming to certify your skills, this plan will guide you step by step through all four key knowledge domains of the NCP-AIO exam:

1. Administration – Learn how to manage multi-GPU systems using Base Command Manager (BCM), Slurm, DCGM, and Fleet Command.

2. Workload Management – Master workload scheduling, multi-tenancy, Kubernetes GPU integration, and monitoring techniques.

3. Installation & Deployment – Gain hands-on experience setting up drivers, container runtimes, GPU operators, and edge AI systems.

4. Troubleshooting & Optimization – Understand how to diagnose system failures, analyze performance, and optimize AI pipelines using Nsight, NCCL, and DCGM.

This plan is structured around the Pomodoro technique to improve focus and prevent burnout, and incorporates Ebbinghaus' forgetting curve principles to enhance long-term retention through spaced review and reflection. Each week targets one major domain and is divided into daily learning goals with specific tasks, tool-based exercises, review prompts, and real-world use cases.

By following this plan:

• You’ll build not only theoretical knowledge but practical fluency with real commands and tools.

• You’ll understand how to troubleshoot live issues and make intelligent decisions under resource constraints.

• You’ll prepare confidently for the NCP-AIO exam while also gaining skills directly applicable to professional environments.

Stay disciplined, take notes, and review strategically. This journey will not only help you earn a certification—it will make you a capable and confident AI infrastructure professional.

WEEK 1: ADMINISTRATION

Weekly Objective:
By the end of this week, you should be able to confidently manage and monitor an AI infrastructure using tools such as Base Command Manager (BCM), Fleet Command, Slurm, MIG, and DCGM. You should be able to configure GPU clusters, assign users, schedule jobs, and monitor GPU health.

Day 1 – Base Command Manager (BCM) Fundamentals

Daily Goal: Understand the architecture, initialization, and interface of BCM.

Tasks:

1. Study what BCM is and where it fits in GPU cluster infrastructure.

2. Learn about bcminit and how it initializes a new node for cluster participation.

3. Explore BCM’s CLI and web interface, including dashboard navigation.

4. Practice simulating node registration and creating a role-based user.

5. Summarize BCM's integration with Slurm.

Pomodoro Sessions:

• 1: Read BCM documentation and workflow diagrams.

• 2: Watch a short BCM demo or run through interface simulation (if sandbox available).

• 3: Attempt CLI usage examples (e.g., bcm system list, bcm user add).

• 4: Summarize key commands and draw the cluster topology.

Review Task (Evening):

• Quick 15-minute review of terminology: bcminit, node, agent, role, integration.

• Flashcard test: What is the difference between a registered node and an active node?

Day 2 – Slurm Integration and Scheduling Basics

Daily Goal: Learn how Slurm works within BCM and how to submit and track jobs.

Tasks:

1. Understand the core components: slurmctld, slurmd, slurm.conf, sacctmgr.

2. Study commands: sbatch, squeue, sinfo.

3. Write and test a sample Slurm job script requesting 1 GPU.

4. Submit a batch job and analyze the result with squeue.

5. Explore how partitions are configured and assigned.

Pomodoro Sessions:

• 1: Read Slurm architecture and role in GPU job scheduling.

• 2: Write your first sbatch script with GPU and memory request.

• 3: Submit job and troubleshoot output using squeue, sacct, cat slurm*.out.

• 4: Record and explain the purpose of each Slurm config term.

Review Task (Evening):

• Re-watch a job submission if possible; quiz yourself on what each command does.

Day 3 – Fleet Command Overview and Remote Management

Daily Goal: Understand how to use Fleet Command to manage edge devices and AI workloads.

Tasks:

1. Learn what Fleet Command is and how it's used in edge AI deployment.

2. Understand how to register a node using a secure token.

3. Simulate deploying a container to an edge device (real or mock).

4. Learn how Fleet enables remote app restart, logs, and status monitoring.

Pomodoro Sessions:

• 1: Watch official NVIDIA or YouTube demo of Fleet Command.

• 2: Simulate (or study) the process of registering a Jetson/Xavier/A100 device.

• 3: Study container lifecycle management and remote control (start/stop/logs).

• 4: Create a written list of Fleet Command's benefits and key use cases.

Review Task (Evening):

• Draw the deployment flow: from Fleet control plane → container → edge node.

Day 4 – MIG: Multi-Instance GPU Architecture

Daily Goal: Learn how to configure and use MIG to partition GPUs into isolated instances.

Tasks:

1. Study MIG use cases: GPU sharing, tenant isolation, workload separation.

2. Use nvidia-smi -mig 1 to enable MIG mode (study output examples).

3. Learn to create MIG instances via nvidia-smi mig -cgi.

4. Assign a MIG instance to a container and verify with docker and nvidia-smi.

5. Understand performance implications of MIG profiles.

Pomodoro Sessions:

• 1: Read MIG whitepaper and review supported profiles (e.g., 1g.10gb, 3g.20gb).

• 2: Watch a CLI-based MIG setup video or walkthrough.

• 3: Practice viewing MIG instances with nvidia-smi, learn device IDs.

• 4: Sketch out MIG logical layout and explain why MIG improves resource efficiency.

Review Task (Evening):

• Write an FAQ: What happens when a container requests a GPU without a MIG assignment?

Day 5 – DCGM: Data Center GPU Manager and Health Monitoring

Daily Goal: Use DCGM to monitor GPU health, track metrics, and export performance data.

Tasks:

1. Learn what DCGM is and how it integrates with Prometheus.

2. Run commands like dcgmi discovery -l, dcgmi health -c, dcgmi stats.

3. Understand metrics: temperature, power draw, memory errors, utilization.

4. Practice building a DCGM health report.

5. (Optional) Configure Prometheus to scrape DCGM-exporter data.

Pomodoro Sessions:

• 1: Study DCGM architecture and how it's deployed (bare metal vs Kubernetes).

• 2: Practice the core CLI commands using dcgmi or nvidia-smi dmon.

• 3: Analyze output logs and simulate a thermal breach alert.

• 4: Create a GPU dashboard mockup: utilization, ECC, power, temp.

Review Task (Evening):

• Recite 5 conditions that would cause GPU health status to change from "Healthy" to "Warning" or "Unhealthy".

Day 6 – Consolidated Hands-On Lab (Integration Practice)

Daily Goal: Simulate an end-to-end workflow from GPU setup to job monitoring.

Tasks:

1. Set up a virtual or simulated GPU cluster using BCM.

2. Register nodes and verify health.

3. Schedule a job using Slurm within BCM (with MIG if possible).

4. Monitor GPU usage and health via DCGM and nvidia-smi.

5. Stop a node, simulate a fault, and observe behavior in BCM or Fleet.

Pomodoro Sessions:

• 1: Review and reconfigure the cluster environment.

• 2: Submit a GPU-bound Slurm job and verify it completes.

• 3: Simulate a GPU overheating case and trigger DCGM alert.

• 4: Document the end-to-end steps and identify failure points.

Review Task (Evening):

• Write a lab report: What worked, what failed, what needs improvement.

Day 7 – Review and Weekly Assessment

Daily Goal: Reinforce all concepts through a structured review and mini exam.

Tasks:

1. Take a 10-question quiz covering BCM, Slurm, Fleet, MIG, and DCGM.

2. Revisit the flashcards and notes from Days 1–6.

3. Draw a concept map showing relationships between components (e.g., BCM ↔ Slurm ↔ DCGM).

4. Identify weak areas and schedule follow-up reviews on Day 9 or Day 14 (based on Ebbinghaus curve).

Pomodoro Sessions:

• 1: Quiz attempt and corrections.

• 2: Flashcard & mindmap creation.

• 3: Mistake analysis and re-read notes.

• 4: Optional challenge: build a 3-minute oral explanation of how an AI job flows through the system (BCM → Slurm → MIG → DCGM).

Review Task (Evening):

• Reflect in a learning journal: What did I learn this week? What tools do I now feel comfortable using?

WEEK 2: WORKLOAD MANAGEMENT

Weekly Objective:
By the end of this week, you should be able to schedule GPU workloads using Kubernetes and Slurm, configure fair resource usage across users or teams, and monitor real-time GPU utilization using Prometheus and Grafana.

Day 8 – Kubernetes GPU Scheduling Basics

Daily Goal: Learn how Kubernetes schedules GPU workloads using the NVIDIA GPU Operator and device plugin.

Tasks:

1. Study how Kubernetes recognizes GPU resources using the NVIDIA device plugin.

2. Understand how to configure GPU limits and requests using the nvidia.com/gpu key.

3. Write a pod YAML that requests one GPU and deploy it.

4. Use kubectl describe to verify scheduling behavior and GPU allocation.

5. Access the container and confirm GPU availability using nvidia-smi.

Pomodoro Sessions:

• 1: Read official docs on the NVIDIA GPU Operator.

• 2: Deploy a sample pod requesting GPU resources.

• 3: Use kubectl logs and exec to enter the pod and validate GPU access.

• 4: Practice modifying YAML files to adjust resource limits and observe changes.

Evening Review Task:

• Write down 3 things that must be true for a pod to access a GPU on Kubernetes.

Day 9 – Advanced Kubernetes Scheduling Techniques

Daily Goal: Learn to control pod placement using nodeSelectors, taints, and affinity rules.

Tasks:

1. Study how nodeSelectors and labels guide pod placement.

2. Add labels to GPU nodes (e.g., gpu=true) using kubectl label nodes.

3. Create a pod spec with node affinity and tolerations.

4. Simulate scheduling conflicts by removing labels or applying taints.

5. Debug and resolve a scheduling failure using kubectl describe.

Pomodoro Sessions:

• 1: Label nodes and review kubectl get nodes --show-labels.

• 2: Write and apply a manifest using affinity and nodeSelector.

• 3: Experiment with taints and tolerations using kubectl taint.

• 4: Summarize scheduling rule precedence and best practices.

Evening Review Task:

• List three reasons a pod might remain in a "Pending" state when requesting a GPU.

Day 10 – Slurm Job Isolation and Scheduling Policies

Daily Goal: Learn how Slurm manages workload distribution, prioritization, and user fairness.

Tasks:

1. Study how partitions, QOS, and job limits are configured in slurm.conf and sacctmgr.

2. Create two partitions (e.g., teamA, teamB) and assign each to different users.

3. Submit jobs with constraints using sbatch and observe resource assignment.

4. Modify job priority or QOS weight and re-submit to observe impact.

5. Use squeue and scontrol show to explore job properties.

Pomodoro Sessions:

• 1: Review the concept of QOS and fair-share scheduling.

• 2: Configure two user accounts and submit jobs under each.

• 3: Record results from changing fair-share weight or priority.

• 4: Write a comparison between Slurm and Kubernetes scheduling.

Evening Review Task:

• Define what a Slurm "partition" is and how it differs from a Kubernetes namespace.

Day 11 – Multi-Tenancy with Namespaces and MIG

Daily Goal: Set up tenant isolation using Kubernetes namespaces and GPU partitioning with MIG.

Tasks:

1. Create two namespaces for team separation using kubectl create namespace.

2. Assign Role-Based Access Control (RBAC) policies for users within each namespace.

3. Enable MIG mode on a GPU and allocate different instances to each team.

4. Deploy separate pods in each namespace and bind them to the correct MIG partition.

5. Confirm isolation by monitoring GPU usage from within the pods.

Pomodoro Sessions:

• 1: Review MIG instance types and valid combinations.

• 2: Practice creating MIG profiles with nvidia-smi mig -cgi.

• 3: Deploy a GPU workload in each namespace and verify it uses the correct MIG slice.

• 4: Document isolation architecture and security concerns.

Evening Review Task:

• Summarize how namespaces and MIG together enforce tenant isolation.

Day 12 – GPU Monitoring with Prometheus and Grafana

Daily Goal: Set up real-time GPU monitoring using Prometheus and visualize metrics using Grafana.

Tasks:

1. Deploy the dcgm-exporter DaemonSet across GPU nodes.

2. Install and configure Prometheus to scrape the DCGM metrics endpoint.

3. Set up Grafana and import a dashboard that displays GPU usage, temperature, memory.

4. Observe and compare metrics during workload runs.

5. Record key GPU metrics and thresholds (e.g., 90% memory usage alert).

Pomodoro Sessions:

• 1: Install and verify the dcgm-exporter pods on each GPU node.

• 2: Configure Prometheus scrape jobs for GPU metrics.

• 3: Create or import a Grafana dashboard showing metrics per GPU and per job.

• 4: List alerts that could indicate a failed or overloaded GPU.

Evening Review Task:

• Note the five most important GPU metrics you would monitor in a production AI cluster.

Day 13 – Logging and Diagnostics for AI Workloads

Daily Goal: Learn how to debug failed workloads and investigate GPU job logs.

Tasks:

1. Trigger a pod failure by misconfiguring GPU requests (e.g., ask for 2 GPUs on a 1-GPU node).

2. Use kubectl logs, kubectl describe, and events to diagnose the issue.

3. Compare Kubernetes logging with Docker log inspection.

4. Review Slurm job logs: slurmctld.log, slurmd.log, and job output files.

5. Write down common GPU-related errors: OOM, "GPU not found", job stuck.

Pomodoro Sessions:

• 1: Simulate errors and failures on purpose.

• 2: Use Kubernetes and Docker commands to analyze what happened.

• 3: Read and summarize job logs in both K8s and Slurm environments.

• 4: Create a personal "Error Codebook" with solutions.

Evening Review Task:

• Practice solving a sample error: Pod stuck in Pending → what steps would you take?

Day 14 – Weekly Review and Test

Daily Goal: Solidify knowledge and evaluate your understanding through active recall and testing.

Tasks:

1. Take a 10-question multiple-choice test covering:

   • Kubernetes GPU scheduling

   • Node affinity rules

   • Slurm partitioning and QOS

   • Prometheus/Grafana monitoring

   • MIG and namespace isolation

2. Revisit missed questions and understand why the correct answer is right.

3. Create a mind map connecting the following elements:
   Pod → Node → GPU → MIG → Monitoring → Logs

4. Record weak points in a journal and plan to review them in Days 16 and 18 based on forgetting curve.

Pomodoro Sessions:

• 1: Complete the quiz and grade yourself.

• 2: Review and analyze each incorrect answer.

• 3: Organize your notes from the week by theme (scheduling, monitoring, isolation).

• 4: Challenge yourself: give a 3-minute spoken explanation of “how GPU workloads are scheduled and monitored in Kubernetes.”

Evening Review Task:

• Reflect: What knowledge or tools do you now feel confident with? Where do you still feel unsure?

WEEK 3: INSTALLATION & DEPLOYMENT

Weekly Objective:
By the end of this week, you should be able to perform complete GPU infrastructure setup. This includes GPU driver installation, container runtime configuration, Kubernetes GPU enablement, BCM/Fleet deployments, and automation using infrastructure-as-code tools.

Day 15 – Installing NVIDIA GPU Drivers and CUDA Toolkit

Daily Goal: Install and verify GPU drivers on Linux, including CUDA integration for GPU acceleration.

Tasks:

1. Understand the role of NVIDIA drivers and when CUDA Toolkit is required.

2. Install the latest stable NVIDIA GPU driver (e.g., nvidia-driver-535) using apt or .run file.

3. Verify GPU recognition with nvidia-smi.

4. Optionally install the CUDA Toolkit (e.g., 12.0) using DEB packages.

5. Reboot system and confirm kernel module loading (lsmod | grep nvidia).

Pomodoro Sessions:

• 1: Read installation guide for your Linux distribution (Ubuntu recommended).

• 2: Execute installation steps and capture any errors.

• 3: Test GPU visibility with nvidia-smi and resolve potential issues.

• 4: Record all steps and common troubleshooting techniques.

Evening Review Task:

• Write a short checklist: post-install tests you must run to confirm GPU driver success.

Day 16 – Setting Up NVIDIA Container Toolkit for Docker

Daily Goal: Configure container runtime to support GPU access inside containers.

Tasks:

1. Learn what the NVIDIA Container Toolkit does and how it connects GPU drivers to containers.

2. Install nvidia-container-toolkit and configure Docker to recognize the nvidia runtime.

3. Validate setup using:
   docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

4. Explore how container runtimes (containerd/CRI-O) are supported.

5. Review alternative options like Podman if time allows.

Pomodoro Sessions:

• 1: Read official NVIDIA documentation on container toolkit installation.

• 2: Execute installation steps and restart Docker service.

• 3: Run GPU containers and test with real CUDA workloads.

• 4: Debug common errors like runtime not found or GPU device not mapped.

Evening Review Task:

• Write a Docker run command from memory that correctly starts a GPU-enabled container.

Day 17 – Installing NVIDIA GPU Operator in Kubernetes

Daily Goal: Use the GPU Operator to automatically manage GPU driver and plugin installation across K8s clusters.

Tasks:

1. Install a minimal Kubernetes cluster (e.g., kubeadm, minikube, or microk8s).

2. Install NVIDIA GPU Operator using Helm or prebuilt manifests.

3. Confirm the following components are running:

   • GPU driver container

   • Device plugin

   • DCGM exporter

4. Deploy a test pod that requests one GPU and validate usage inside the container.

5. Use kubectl get pods -A and describe to debug any issues.

Pomodoro Sessions:

• 1: Read about GPU Operator architecture and how it simplifies deployment.

• 2: Execute deployment on your cluster and monitor pod status.

• 3: Deploy GPU test pod and inspect runtime behavior.

• 4: Diagram how each Operator component contributes to GPU functionality.

Evening Review Task:

• Explain why using the Operator is better than manually installing drivers on GPU nodes.

Day 18 – Installing and Configuring Base Command Manager (BCM)

Daily Goal: Understand how to install and register nodes into BCM for centralized management.

Tasks:

1. Provision a base OS (Ubuntu or CentOS) for BCM installation.

2. Install the BCM agent using the official installer.

3. Use the BCM CLI to register the node into the cluster.

4. Connect Slurm to BCM and verify job scheduler integration.

5. Practice running commands like bcmsystem status, bcm user list, and check node visibility.

Pomodoro Sessions:

• 1: Read BCM installation prerequisites and role requirements.

• 2: Install and configure BCM Agent.

• 3: Register the node and verify on BCM web interface.

• 4: Attempt scheduling a job and tracking it through BCM.

Evening Review Task:

• Write the complete registration flow from new server to active node in BCM.

Day 19 – Fleet Command Deployment for Edge AI

Daily Goal: Learn to deploy and manage containerized AI apps on edge devices using Fleet Command.

Tasks:

1. Understand how Fleet Command fits into NVIDIA’s cloud-edge architecture.

2. Simulate edge registration using token-based provisioning.

3. Deploy a test container (e.g., Jetson-compatible inference model).

4. Restart and stop containers from the web dashboard.

5. Use logs and monitoring tools in Fleet to verify deployment success.

Pomodoro Sessions:

• 1: Read the lifecycle flow of Fleet Command deployment.

• 2: Complete simulated deployment using demo tokens or documentation.

• 3: Explore use cases in smart retail, logistics, or healthcare AI.

• 4: Diagram cloud-to-edge deployment and feedback loop.

Evening Review Task:

• List the steps for securely provisioning an edge device into Fleet Command.

Day 20 – Introduction to DOCA and Magnum IO

Daily Goal: Understand acceleration frameworks for networking and data I/O in large-scale training.

Tasks:

1. Study what DOCA is and how it supports NVIDIA BlueField DPUs.

2. Review sample use cases: network security, packet inspection, traffic isolation.

3. Learn what Magnum IO does and why it matters in distributed training.

4. Review features like NCCL, UCX, GPUDirect RDMA.

5. Watch a demo or read a blog where Magnum IO boosts I/O efficiency.

Pomodoro Sessions:

• 1: Read the DOCA whitepaper or product page.

• 2: Study Magnum IO stack components.

• 3: Sketch the data path for distributed GPU training with and without Magnum IO.

• 4: Write a use case summary comparing traditional I/O with GPU-optimized stack.

Evening Review Task:

• Write down two practical use cases each for DOCA and Magnum IO.

Day 21 – Deployment Automation and Weekly Quiz

Daily Goal: Use automation tools to configure GPU environments and validate all skills learned during the week.

Tasks:

1. Study basic Ansible playbooks or Terraform scripts for automated provisioning.

2. Simulate driver installation, container runtime setup, and Kubernetes GPU plugin configuration using code.

3. Review Helm usage to install GPU Operator or other components.

4. Take a 10-question quiz on this week's topics: GPU driver, Docker toolkit, Kubernetes deployment, BCM, Fleet.

5. Review mistakes and plan to re-test in a few days following Ebbinghaus intervals.

Pomodoro Sessions:

• 1: Read sample Ansible playbooks related to GPU provisioning.

• 2: Write a basic Terraform or Helm config to deploy the GPU Operator.

• 3: Execute your automation and fix any errors.

• 4: Take the quiz and write out the explanations for each correct answer.

Evening Review Task:

• Create a full checklist for deploying a new GPU server into a production AI cluster.

WEEK 4: TROUBLESHOOTING & OPTIMIZATION

Weekly Objective:
By the end of this week, you should be confident in using GPU monitoring tools, diagnosing performance issues, debugging AI job failures, and optimizing training performance using profiling tools like Nsight, NCCL, and DCGM.

Day 22 – GPU Monitoring Using DCGM and nvidia-smi

Daily Goal: Learn to monitor and diagnose GPU health and performance in real time.

Tasks:

1. Use nvidia-smi to observe real-time GPU state: memory usage, temperature, active processes.

2. Run nvidia-smi dmon to monitor GPU over time.

3. Study DCGM components and install the CLI tools.

4. Use dcgmi discovery -l, dcgmi health -c, and dcgmi stats to assess GPU health.

5. Trigger and investigate a GPU health alert (e.g., simulate a temp spike or ECC warning).

Pomodoro Sessions:

• 1: Practice basic nvidia-smi commands and explore all flags.

• 2: Install and try dcgmi on a test system or simulation.

• 3: Record GPU metrics under load and idle conditions.

• 4: Document what each metric means and what thresholds indicate danger.

Evening Review Task:

• Write down three reasons why DCGM is preferred over nvidia-smi in large-scale environments.

Day 23 – Diagnosing Docker and Container Failures

Daily Goal: Identify and fix container issues that prevent GPU access or cause job crashes.

Tasks:

1. Run a GPU container using --gpus all and verify access.

2. Remove the runtime or misconfigure the container, then troubleshoot the errors.

3. Simulate a common issue (e.g., missing NVIDIA runtime, container cannot find GPU).

4. Use docker info | grep -i runtime and logs to trace the issue.

5. Study how MIG misconfigurations cause GPU devices to appear missing or busy.

Pomodoro Sessions:

• 1: Test multiple valid and invalid container run scenarios.

• 2: Practice using Docker logs and inspecting container status.

• 3: Explore what happens when MIG devices are misassigned.

• 4: Create a container GPU access checklist.

Evening Review Task:

• From memory, list all the Docker options and system settings needed to run a GPU container successfully.

Day 24 – Kubernetes Debugging Techniques

Daily Goal: Debug GPU workload failures in Kubernetes using pod logs, events, and node conditions.

Tasks:

1. Deploy a GPU pod and validate that it starts successfully.

2. Purposely cause it to fail (e.g., invalid resource request or wrong nodeSelector).

3. Use kubectl logs, describe, and get events to trace the root cause.

4. Investigate common failure messages: OOMKilled, CrashLoopBackOff, Unschedulable.

5. Inspect GPU plugin DaemonSets to ensure node-level GPU resources are healthy.

Pomodoro Sessions:

• 1: Practice interpreting kubectl describe outputs and error codes.

• 2: Study the impact of node taints, missing device plugins, and resource mislabels.

• 3: Create a Kubernetes GPU diagnostic flowchart.

• 4: Compare pod-level versus node-level error investigation.

Evening Review Task:

• Write a scenario where a pod stays pending due to GPU-related issues, and explain how to fix it.

Day 25 – Using Nsight Systems for Profiling AI Workloads

Daily Goal: Use Nsight Systems to analyze bottlenecks in training workflows.

Tasks:

1. Install Nsight Systems and run a profile on a small AI training job.

2. Analyze kernel launch delays, memory transfers, and CPU bottlenecks.

3. Identify long durations in data loading or I/O stalls in timeline view.

4. Practice filtering timeline results to focus on GPU compute activity.

Pomodoro Sessions:

• 1: Review how Nsight Systems collects timeline data.

• 2: Run a sample workload with nsys profile.

• 3: Interpret slowdowns in data movement and compute overlap.

• 4: Record 2–3 concrete performance improvements you could make.

Evening Review Task:

• Write down what a spike in CPU wait time vs GPU wait time indicates in a training job.

Day 26 – Using Nsight Compute for GPU Kernel Optimization

Daily Goal: Analyze individual CUDA kernels for memory inefficiencies or underutilized compute.

Tasks:

1. Use nv-nsight-cu-cli or the GUI version to profile a specific CUDA kernel.

2. Study occupancy, memory coalescing, and warp divergence.

3. Identify one inefficient kernel and analyze its cause.

4. Compare how small vs large batch sizes affect GPU kernel usage.

Pomodoro Sessions:

• 1: Read Nsight Compute reports and what each metric means.

• 2: Run profiling on at least two different AI model kernels.

• 3: Compare kernel execution under different batch sizes or precisions.

• 4: Write a kernel optimization plan for a sample job.

Evening Review Task:

• List three kernel metrics that strongly affect GPU throughput.

Day 27 – Troubleshooting Interconnect and I/O Bottlenecks

Daily Goal: Use topology and bandwidth tools to analyze NVLink, PCIe, and NCCL communication.

Tasks:

1. Run nvidia-smi topo -m to view NVLink/NVSwitch interconnect layout.

2. Use nccl-tests (e.g., all_reduce_perf) to evaluate bandwidth across GPUs.

3. Simulate a job that is communication-bound and observe performance hit.

4. Study the difference in performance between NVLink and PCIe-based inter-GPU transfers.

Pomodoro Sessions:

• 1: Practice visualizing GPU-to-GPU topologies.

• 2: Use NCCL benchmarks to test GPU bandwidth.

• 3: Profile distributed training jobs and note time spent on communication ops.

• 4: Summarize when interconnect optimization is critical.

Evening Review Task:

• Write the complete path data takes when moving from one GPU to another in a DGX system using NVSwitch.

Day 28 – Final Practice, Quiz, and Knowledge Consolidation

Daily Goal: Bring everything together and assess your readiness for the exam.

Tasks:

1. Take a 40-question mock exam covering all 4 domains.

2. Identify which areas are still weak and schedule a final revision day.

3. Review logs, tools, and command-line utilities you've used throughout the course.

4. Consolidate notes into a one-page “cheat sheet” per domain (Administration, Workload Management, Installation & Deployment, Troubleshooting & Optimization).

5. Use flashcards to rehearse terminology, commands, and system behaviors.

Pomodoro Sessions:

• 1: Complete the mock test and record score.

• 2: Review each incorrect answer and look up detailed explanations.

• 3: Reorganize notes into condensed visual mind maps.

• 4: Deliver a 5-minute self-presentation covering a full AI pipeline: provisioning → workload deployment → job monitoring → troubleshooting.

Evening Review Task:

• Reflect in your learning journal: What have you mastered? What needs reinforcement? What tool or concept surprised you most?

---