Skip to main content

Lab 10: GPU Topology-Aware Scheduling on Fake GPUs

IntermediateDuration: about 45 minutesEnvironment: macOS (OrbStack) ยท Linux (Ubuntu + kind) ยท no GPU requiredCost: freeVerified: 2026-07-20By: @maishivamhoo123

This lab walks you through simulating an asymmetric PCIe topology on a single-node local cluster using nvml-mock and HAMi. You'll enable HAMi's topology-aware scheduler, inject custom connectivity scores to isolate one GPU, and then verify that a multi-GPU request avoids the isolated GPU while a single-GPU request picks it. No physical GPUs are needed - everything runs inside a local Kubernetes cluster.

What You'll Getโ€‹

After completing this lab, you will have:

  • A local cluster with nvml-mock simulating 8 A100 GPUs (80 virtual slots after HAMi slices each GPU into 10)
  • HAMi built from a verified commit and installed with topology-aware scheduling (gpuSchedulerPolicy=topology-aware)
  • A node annotation (hami.io/node-nvidia-score) that defines a custom topology where GPU7 is poorly connected to all other GPUs
  • Proof that the scheduler avoids GPU7 when allocating 2 GPUs to a single Pod (multi-GPU request), and picks GPU7 for a single-GPU request - both behaviors come from the same topology-aware policy, with no extra flag required
  • Visibility into the scheduler's topology decisions via its own logs
note

The fake GPU topology scores here are synthetic - nvml-mock reports symmetric connectivity by default, and we overwrite the node annotation ourselves to create an artificial "worst-connected" GPU. This lab validates the scheduler's logic against a known topology, not real PCIe/NVLink measurement.

Score direction matters. In HAMi's actual scheduling code (pkg/device/nvidia/device.go), a higher pairwise score means better connectivity (like NVLink), and a lower score means worse. Multi-GPU requests pick the combination with the highest total score (via computeBestCombination); single-GPU requests pick the device with the lowest score (via computeWorstSingleCard) - both paths are gated by the same needTopology check in Fit(), driven entirely by gpuSchedulerPolicy=topology-aware. There is no separate flag to enable single-GPU scoring; it's on by default the moment the policy is set. To make GPU7 the worst-connected device, we set its scores below the 50 baseline, not above it.

Installation Overviewโ€‹

The entire lab consists of 7 steps:

Topology-Aware Scheduling Lab Overview
StepPurposeWhat It Solves
Set Up & Verify EnvironmentCreate/verify cluster, check toolsEnsure a Kubernetes cluster is available
Build nvml-mockSimulate 8 A100 GPUsGives the device plugin an NVML topology to read
Build & Install HAMiDeploy scheduler with topology-aware policyEnables the scheduler to consider connectivity scores for both single- and multi-GPU requests
Isolate GPU7Freeze the device plugin and overwrite the topology annotationCreates a known-bad-connectivity GPU to test against
Raise Scheduler Log VerbosityBump -v=6 on the schedulerReveals the best device combination / worst device topology log lines
Verify Scheduling BehaviorMulti-GPU Pod + single-GPU PodConfirms avoid/pick behavior matches the injected topology
Observe Per-Device ScoresScheduler logs at -v=6Shows the actual best device combination / worst device lines

Prerequisitesโ€‹

  • macOS, Intel or Apple Silicon
  • OrbStack installed with built-in Kubernetes enabled
  • docker, git, python3
  • Access to GitHub, GHCR, and the HAMi Helm repository
  • At least 8 GB of free memory and 4 CPU cores available
Why OrbStack?

OrbStack comes with built-in Kubernetes (based on k3s), so there's no need to install kind or Docker Desktop separately. It uses fewer resources, starts faster, and is the preferred choice for local labs on macOS.

Check Helm:

helm version

If Helm is not installed:

brew install helm

Step 1: Set Up and Verify Local Environmentโ€‹

Why: We need a working Kubernetes cluster to deploy nvml-mock and HAMi.

OrbStack's Kubernetes starts automatically once enabled in the OrbStack UI. Verify the cluster is ready:

kubectl version

Example output:

Client Version: v1.33.9
Kustomize Version: v5.6.0
Server Version: v1.33.9+orb1
note

The +orb1 suffix in Server Version identifies OrbStack's built-in Kubernetes distribution.

Set the NODE_NAME Variableโ€‹

The rest of this lab uses a NODE_NAME shell variable so commands don't hard-code the node name. Set it at the beginning of each new terminal session because the variable is lost when you close the shell.

NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
echo "NODE_NAME=${NODE_NAME}"
Most remaining steps are the same on both platforms

From here on, most commands work the same on macOS and Linux. A couple of steps still differ - Step 2.3 and Step 3.1 each call out an extra kind load command needed only on Linux. Everything else is identical, aside from the node name you'll see in example outputs.

Step 2: Build and Deploy nvml-mock (8 Simulated A100 GPUs)โ€‹

Why: nvml-mock provides fake GPU devices and NVML topology data so HAMi can discover and score them.

2.1 Clone the NVIDIA Test Infrastructure Repositoryโ€‹

git clone https://github.com/NVIDIA/k8s-test-infra.git
cd k8s-test-infra

2.2 Build the Mock Docker Imageโ€‹

docker build -t nvml-mock:local -f deployments/nvml-mock/Dockerfile .

2.3 Load the Image into the Cluster and Installโ€‹

On macOS with OrbStack, the local Docker image is directly accessible to the cluster โ€” no kind load is needed.

helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
  --set image.repository=nvml-mock \
  --set image.tag=local \
  --wait --timeout 120s

2.4 Verify the GPU Label Is Presentโ€‹

kubectl get node ${NODE_NAME} -o custom-columns=NAME:.metadata.name,GPU:.metadata.labels.nvidia\\.com/gpu\\.present
NAME                     GPU
topo-lab-control-plane   true

The GPU column shows true, meaning nvml-mock has registered simulated GPU devices on the node.

Step 3: Build HAMi from Main and Install with Topology-Aware Schedulingโ€‹

Why: HAMi's scheduler replaces the default Kubernetes scheduler for GPU pods and can make topology-aware decisions when the topology-aware policy is enabled.

3.1 Clone HAMi and Check Out the Verified Commitโ€‹

The lab was tested against HAMi main branch as of 2026-07-14. If your current main doesn't work as expected, use the exact commit a1b418c:

cd ~
git clone https://github.com/Project-HAMi/HAMi.git
cd HAMi
git log --until=2026-07-14 --oneline -1
git checkout a1b418c
git submodule update --init --recursive
docker build -t hami:local -f docker/Dockerfile .
# Image is already available locally, no need to load

3.2 Install HAMi with the Topology-Aware Policyโ€‹

Read the Kubernetes server version straight from the running cluster so the scheduler's bundled kube-scheduler binary stays aligned with whatever API server you actually have. OrbStack reports something like v1.33.9+orb1 while kind reports v1.35.0, so a single hardcoded tag would only ever be correct on one platform:

K8S_VERSION=$(kubectl version -o json | python3 -c "import json,sys; print(json.load(sys.stdin)['serverVersion']['gitVersion'].split('+')[0])")
echo "K8S_VERSION=${K8S_VERSION}"
helm install hami ./charts/hami \
  -n kube-system \
  --set devicePlugin.image.repository=hami \
  --set devicePlugin.image.tag=local \
  --set scheduler.image.repository=hami \
  --set scheduler.image.tag=local \
  --set devicePlugin.nvidiaDriverRoot=/var/lib/nvml-mock/driver \
  --set scheduler.kubeScheduler.imageTag=${K8S_VERSION} \
  --set scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy=topology-aware

gpuSchedulerPolicy=topology-aware tells the scheduler to use pairwise connectivity scores from the node annotation when selecting GPUs. This single setting covers both multi-GPU and single-GPU requests - HAMi's Fit() function (pkg/device/nvidia/device.go) checks this policy once, then branches internally: multi-GPU requests go through computeBestCombination(), single-GPU requests go through computeWorstSingleCard(). There's no separate flag to turn on for single-GPU scoring. scheduler.kubeScheduler.imageTag is read from the live cluster rather than hardcoded, since OrbStack and kind ship different Kubernetes versions, and this keeps the scheduler's embedded kube-scheduler binary aligned with whatever API server is actually running.

3.3 Label the Node and Configure NVML Discoveryโ€‹

Confirm NODE_NAME is set before running this

If you opened a new terminal since Step 1, or NODE_NAME otherwise ended up empty, the label command below silently degrades instead of clearly failing on the node โ€” but the label itself is what breaks:

kubectl label node ${NODE_NAME} gpu=on --overwrite
error: resource(s) were provided, but no name was specified

That error means ${NODE_NAME} expanded to nothing, so the command became kubectl label node gpu=on --overwrite with no node name at all. The node never gets the gpu=on label, and everything downstream fails quietly:

  • If hami-device-plugin's DaemonSet has a nodeSelector requiring gpu=on, it now matches zero nodes.
  • kubectl rollout restart / rollout status on a DaemonSet with zero desired pods reports "successfully rolled out" โ€” there's nothing to roll out, so this gives no indication anything is wrong.
  • kubectl get pods | grep hami-device-plugin returns nothing at all (not Pending, not CrashLoopBackOff โ€” no pod exists anywhere).
  • Every step after this that depends on the device plugin running (topology annotation, GPU capacity, actual pod scheduling) will fail later with confusing, disconnected errors.

Before running the block below, confirm NODE_NAME is non-empty:

echo "NODE_NAME=${NODE_NAME}"
NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
kubectl label node ${NODE_NAME} gpu=on --overwrite
kubectl -n kube-system set env daemonset/hami-device-plugin -c device-plugin DEVICE_DISCOVERY_STRATEGY=nvml
kubectl -n kube-system rollout restart daemonset/hami-device-plugin
kubectl -n kube-system rollout status daemonset/hami-device-plugin --timeout=120s

Confirm the pod actually exists before moving on โ€” a clean rollout status is not proof by itself:

kubectl -n kube-system get pods -o wide | grep hami-device-plugin

If this returns no output, stop and go back to the NODE_NAME / label check above rather than continuing to Step 3.4 โ€” there is no pod to become Running, so waiting longer won't help.

Stop here if this times out

If you see error: timed out waiting for the condition, the device plugin may take longer than 120 seconds to start. Do not continue until the pod is Running and 1/1 Ready.

You can check the pod status with:

kubectl -n kube-system get pods -o wide | grep hami-device-plugin
kubectl -n kube-system describe pod <hami-device-plugin-pod-name>
kubectl -n kube-system logs <hami-device-plugin-pod-name> -c device-plugin --tail=200

Common issues:

  1. Pod stuck Pending โ€“ check for nodeSelector/affinity/taint mismatches. Ensure the node has the labels nvidia.com/gpu.present=true (set by nvml-mock) and gpu=on (set above).

  2. device-plugin container CrashLoopBackOff โ€“ the device plugin can't find the NVML mock library. Verify the path used in devicePlugin.nvidiaDriverRoot matches the actual path nvml-mock installs libnvidia-ml.so.* to (usually /var/lib/nvml-mock/driver). See the nvml-mock pod logs or NOTES output for the correct path.

  3. No pod at all (not even Pending) โ€“ this is the NODE_NAME failure mode above, not a timeout. Check that the node actually carries the gpu=on label with kubectl get node ${NODE_NAME} --show-labels | grep gpu=on.

  4. vgpu-monitor container restarting, pod shows 1/2 โ€“ this is expected and harmless in this lab, not a real failure. vgpu-monitor watches real container runtimes (/run/docker, /run/containerd) for actual GPU-using workloads to report on; since nvml-mock has no real GPU workloads for it to monitor, it exits cleanly (Exit Code: 0, Reason: Completed) and kubelet restarts it, which can trigger a BackOff warning purely from restarting quickly. What actually matters for this lab is the device-plugin container specifically โ€” check its readiness on its own:

    kubectl -n kube-system get pod <hami-device-plugin-pod-name> -o jsonpath='{range .status.containerStatuses[*]}{.name}{": ready="}{.ready}{"\n"}{end}'

    If device-plugin: ready=true, you're fine to proceed even if vgpu-monitor shows restarts and the pod's overall READY column reads 1/2. This same non-critical timeout can resurface at the rollout status in Step 4.4 โ€” same cause, same fix: check device-plugin specifically, not the whole pod's ready count.

Once device-plugin is ready, you can re-run the rollout status command or simply proceed to the next step.

3.4 Check the Node's GPU Capacityโ€‹

kubectl describe node ${NODE_NAME} | grep nvidia.com/gpu
nvidia.com/gpu: 80

80 comes from 8 fake GPUs sliced into 10 vGPU units each by HAMi.

Step 4: Create a Custom Topology that Isolates GPU7โ€‹

Why: By default, all GPU pairs have the same connectivity score (50). We'll manually craft a topology where GPU7 has a very low score (5) with every other GPU, then freeze the device plugin to prevent it from overwriting our custom values. This makes GPU7 the "worst-connected" device.

This step is required - it's easy to end up with an empty annotation without it

Setting gpuSchedulerPolicy=topology-aware at Helm install time (Step 3.2) only turns on the scheduler's side of topology awareness. It does not make the device plugin compute or publish topology scores. That's a separate switch: the ENABLE_TOPOLOGY_SCORE environment variable on the hami-device-plugin DaemonSet, and it defaults to false/off when unset. If you skip 4.1 below, hami.io/node-nvidia-score will be an empty string โ€” and the Python scripts in this step will fail with JSONDecodeError. Capacity registration and topology-score publishing are different code paths; one working doesn't imply the other is on.

4.1 Enable Topology Scoring on the Device Pluginโ€‹

kubectl -n kube-system set env daemonset/hami-device-plugin -c device-plugin ENABLE_TOPOLOGY_SCORE=true
kubectl -n kube-system rollout restart daemonset/hami-device-plugin
kubectl -n kube-system rollout status daemonset/hami-device-plugin --timeout=120s

Wait past one registration cycle (the device plugin patches the annotation every 30 seconds), then check if the annotation is populated:

sleep 40
kubectl get node ${NODE_NAME} -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-score}'

This should print a JSON array of 8 objects (one per fake GPU), each with a uuid and a score map.

  • If you see a JSON array: the annotation is present. Proceed to 4.2.
  • If it prints nothing or fails with an error: the annotation is empty. In that case, we'll manually create a default topology file using the known UUIDs. Follow the fallback instructions inside 4.2.

If the device-plugin pod from Step 3.3 never actually started (check with kubectl -n kube-system get pods | grep hami-device-plugin), an empty annotation here is expected โ€” there's nothing running to populate it. Go back and fix Step 3.3 first rather than proceeding straight to the manual fallback; the fallback works either way, but it's worth knowing which case you're in.

4.2 Obtain the Current Topology (or Create a Default)โ€‹

First, try to pull the annotation from the node:

kubectl get node ${NODE_NAME} -o jsonpath="{.metadata.annotations.hami\.io/node-nvidia-score}" > /tmp/orig-topo.json

Now test if the file contains valid JSON:

python3 -c "import json; d=json.load(open('/tmp/orig-topo.json')); print(f'Devices: {len(d)}')"

If the command succeeds and prints Devices: 8, you can skip the fallback below and continue with the GPU7 UUID extraction.

If the command fails (e.g., JSONDecodeError: Expecting value), the file is empty. Use the following manual fallback to create a default topology.

Don't re-run the pull command after this

Once you've written the fallback file below, do not re-run the kubectl get node ... > /tmp/orig-topo.json command from earlier on this page. If the real annotation is still empty (e.g. the device plugin still isn't running), that command will silently truncate /tmp/orig-topo.json back to 0 bytes, overwriting the good fallback data you're about to create. This is exactly what produces JSONDecodeError: Expecting value: line 1 column 1 (char 0) on every subsequent step, including a blank GPU7_UUID.

If you need to check whether the real annotation has since appeared, redirect to a different file first (e.g. /tmp/check-topo.json) and inspect that, rather than overwriting /tmp/orig-topo.json directly:

kubectl get node ${NODE_NAME} -o jsonpath="{.metadata.annotations.hami\.io/node-nvidia-score}" > /tmp/check-topo.json
python3 -c "import json; d=json.load(open('/tmp/check-topo.json')); print(f'Devices: {len(d)}')"
# only if this succeeds, copy it over: cp /tmp/check-topo.json /tmp/orig-topo.json

Fallback: Create Default Topology Manuallyโ€‹

The nvml-mock a100 profile generates GPUs with UUIDs starting with GPU-12345678-1234-1234-1234-12345678 and ending with the device index (0โ€“7). We'll use these to build the topology file.

cat > /tmp/orig-topo.json <<'ENDJSON'
[
  {"uuid":"GPU-12345678-1234-1234-1234-123456780000","score":{"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780001","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780002","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780003","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780004","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780005","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780006":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780006","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780007":50}},
  {"uuid":"GPU-12345678-1234-1234-1234-123456780007","score":{"GPU-12345678-1234-1234-1234-123456780000":50,"GPU-12345678-1234-1234-1234-123456780001":50,"GPU-12345678-1234-1234-1234-123456780002":50,"GPU-12345678-1234-1234-1234-123456780003":50,"GPU-12345678-1234-1234-1234-123456780004":50,"GPU-12345678-1234-1234-1234-123456780005":50,"GPU-12345678-1234-1234-1234-123456780006":50}}
]
ENDJSON

Verify the file is valid:

python3 -c "import json; d=json.load(open('/tmp/orig-topo.json')); print(f'Devices: {len(d)}'); print('GPU7 UUID:', [x['uuid'] for x in d if x['uuid'].endswith('780007')][0])"

Expected output:

Devices: 8
GPU7 UUID: GPU-12345678-1234-1234-1234-123456780007

4.2.1 Locate GPU7 UUIDโ€‹

Now extract the UUID of GPU7 (whether from the real annotation or the manually-created file):

GPU7_UUID=$(python3 -c "
import json
data = json.load(open('/tmp/orig-topo.json'))
for d in data:
    if d['uuid'].endswith('780007'):
        print(d['uuid'])
        break
")
echo "GPU7 UUID = $GPU7_UUID"
Stop if this is empty โ€” do not proceed to 4.3

If GPU7_UUID prints as empty (GPU7 UUID = with nothing after the equals sign), do not continue to Step 4.3. This most commonly happens when $GPU7_UUID was set in a previous shell session and this is a fresh terminal โ€” the variable simply doesn't exist yet here, $GPU7_UUID silently expands to an empty string in the next command, and Python happily accepts gpu7 = '' without raising any error.

The consequence is severe and easy to miss: the score-generation script in 4.3 will run to completion, produce a custom-topo.json that looks plausible, apply cleanly, and pass every syntax check โ€” but it will never actually touch any real GPU's scores. Instead it silently adds a bogus '' key to every device's score map. The annotation check in 4.6 will look almost right (every real device still shows the old baseline of 50) except for one stray extra key: '': 5. If you see that stray empty-string key anywhere in your topology JSON, this is what happened โ€” go back and re-run the extraction command above in the current shell.

Add this hard stop before moving on:

if [ -z "$GPU7_UUID" ]; then
  echo "ERROR: GPU7_UUID is empty. Re-run the extraction command above in this shell before continuing." >&2
  exit 1
else
  echo "OK: GPU7_UUID is set to $GPU7_UUID"
fi
Match on the UUID suffix, not array position

The array order is not sorted by device index. Always match on the UUID suffix (...78000N for device index N). Never assume data[7] is GPU7.

4.3 Generate a Modified Topology with GPU7 Isolated (Score = 5)โ€‹

python3 -c "
import json
data = json.load(open('/tmp/orig-topo.json'))
gpu7 = '$GPU7_UUID'
for entry in data:
    if entry['uuid'] == gpu7:
        for other in data:
            if other['uuid'] != gpu7:
                entry['score'][other['uuid']] = 5
    else:
        entry['score'][gpu7] = 5
print(json.dumps(data))
" > /tmp/custom-topo.json

Default scores are 50. Higher = better connectivity. Setting GPU7's pairs to 5 makes it the worst-connected: multi-GPU combos involving GPU7 have the lowest total score and are avoided; single-GPU picks the lowest total score, which will be GPU7.

4.4 Freeze the Device Plugin So It Can't Overwrite Your Custom Scoresโ€‹

Now flip the same env var back to false โ€” this stops the device plugin from recomputing and overwriting the manual scores:

kubectl -n kube-system set env daemonset/hami-device-plugin -c device-plugin ENABLE_TOPOLOGY_SCORE=false
kubectl -n kube-system rollout restart daemonset/hami-device-plugin
kubectl -n kube-system rollout status daemonset/hami-device-plugin --timeout=120s
Timeout is non-critical

If this rollout status times out, it's almost always the same harmless vgpu-monitor container restart cycle described in Step 3.3, not a real problem โ€” vgpu-monitor has nothing to monitor in this fake-GPU lab, so it exits cleanly and restarts, which can keep the pod's overall READY count below the daemonset's expected total. What matters is that device-plugin itself is ready and the env var took effect:

kubectl -n kube-system get pods -o wide | grep hami-device-plugin
kubectl -n kube-system get pod <hami-device-plugin-pod-name> -o jsonpath='{range .status.containerStatuses[*]}{.name}{": ready="}{.ready}{"\n"}{end}'

If device-plugin: ready=true, the freeze is effective and you can continue with Step 4.5 regardless of what vgpu-monitor is doing.

4.5 Apply the Custom Annotationโ€‹

Important: The NODE_NAME variable must still be set. If you've opened a new terminal, re-run:

NODE_NAME=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')

Then apply the annotation:

kubectl annotate node ${NODE_NAME} hami.io/node-nvidia-score="$(cat /tmp/custom-topo.json)" --overwrite

Expected output: node/<NODE_NAME> annotated.

4.6 Verify that GPU7 Now Has a Low Score with All Other GPUsโ€‹

Wait past one registration cycle (30s) to confirm the freeze held:

sleep 40
kubectl get node ${NODE_NAME} -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-score}' | python3 -c "
import sys, json
data = json.load(sys.stdin)
for d in data:
    if d['uuid'].endswith('780007'):
        print(d['score'])
        break
"

Expected: a dictionary where every other GPU UUID maps to 5, not 50. Example:

{'GPU-12345678-...-780000': 5, 'GPU-12345678-...-780001': 5, ...}

4.7 Restart the Scheduler So It Picks Up the New Topologyโ€‹

kubectl -n kube-system rollout restart deployment hami-scheduler
kubectl -n kube-system rollout status deployment hami-scheduler --timeout=60s
kubectl delete pod multi-gpu single-gpu --ignore-not-found

Step 5: Raise Scheduler Log Verbosityโ€‹

Why: Single-GPU topology scoring needs no extra flag to switch on - it's already active as soon as gpuSchedulerPolicy=topology-aware was set back in Step 3.2. Raising verbosity to -v=6 exposes the best device combination / worst device log lines that prove the scheduler used your custom scores.

5.1 Raise Scheduler Log Verbosity to 6โ€‹

kubectl -n kube-system get deployment hami-scheduler -o json > /tmp/hami-scheduler.json
python3 -c "
import json
d = json.load(open('/tmp/hami-scheduler.json'))
for c in d['spec']['template']['spec']['containers']:
    cmd = c.get('command') or []
    for i, a in enumerate(cmd):
        if a.startswith('-v='):
            cmd[i] = '-v=6'
json.dump(d, open('/tmp/hami-scheduler.json', 'w'))
"
kubectl apply -f /tmp/hami-scheduler.json
kubectl -n kube-system rollout status deployment hami-scheduler --timeout=60s
kubectl -n kube-system get deployment hami-scheduler -o yaml | grep -- '-v='

You should see -v=6 for both the kube-scheduler and vgpu-scheduler-extender containers.

Step 6: Verification Testsโ€‹

Why: We run a pod that requests 2 GPUs (should avoid GPU7) and another that requests 1 GPU (should pick GPU7). The scheduler's logs and pod annotations confirm the behavior.

6.1 Multi-GPU Pod - Must Avoid GPU7โ€‹

Write the Pod manifest:

cat > multi-gpu-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: multi-gpu
spec:
  containers:
    - name: sleep
      image: busybox
      command: ["sleep", "3600"]
      env:
        - name: CUDA_DISABLE_CONTROL
          value: "true"
      resources:
        limits:
          nvidia.com/gpu: 2
EOF

We use busybox rather than a CUDA image because nvml-mock and HAMi are only simulating GPU scheduling and accounting here. CUDA_DISABLE_CONTROL: "true" keeps HAMi's in-container control layer from trying to intercept a real driver that doesn't exist.

Review and apply:

kubectl apply -f multi-gpu-pod.yaml

Wait for it to run, then check which GPUs it landed on:

kubectl get pod multi-gpu -w   # wait for Running, then Ctrl+C
kubectl get pod multi-gpu -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}'

Expected output: Should list two GPUs with UUIDs that do not end in 780007. Example: GPU-...780001,GPU-...780000.

Check the scheduler log:

kubectl -n kube-system logs deployment/hami-scheduler -c vgpu-scheduler-extender | grep "best device combination" | tail -1

Expected output (indices โ‰  7):

I0717 01:51:55.073459       1 device.go:859] "device allocate success" pod="default/multi-gpu" best device combination={"NVIDIA":[{"Idx":1,"UUID":"GPU-12345678-...-780001",...},{"Idx":0,"UUID":"GPU-12345678-...-780000",...}]}

6.2 Single-GPU Pod - Should Pick GPU7โ€‹

Write the Pod manifest:

cat > single-gpu-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: single-gpu
spec:
  containers:
    - name: sleep
      image: busybox
      command: ["sleep", "3600"]
      env:
        - name: CUDA_DISABLE_CONTROL
          value: "true"
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

Review and apply:

kubectl apply -f single-gpu-pod.yaml

Wait for it to run, then check the allocated GPU:

kubectl get pod single-gpu -w   # wait for Running, then Ctrl+C
kubectl describe pod single-gpu | grep vgpu-devices-allocated

Expected output: hami.io/vgpu-devices-allocated: GPU-...780007.

Check the scheduler log:

kubectl -n kube-system logs deployment/hami-scheduler -c vgpu-scheduler-extender | grep "worst device" | tail -1

Expected output (Idx=7):

I0717 01:52:XX.XXXXXX       1 device.go:853] "device allocate success" pod="default/single-gpu" worst device=[{"Idx":7,"UUID":"GPU-12345678-...-780007",...}]

Step 7: (Optional) Observe the Scheduler's Topology Decisionsโ€‹

Why: The best device combination / worst device log lines are the direct proof that the scheduler used your custom topology scores.

Grep both lines at once:

kubectl -n kube-system logs deployment/hami-scheduler -c vgpu-scheduler-extender | grep -E "best device combination|worst device"

You should see one line for the multi-GPU pod (best combination) and one for the single-GPU pod (worst device), with GPU7 avoided and chosen respectively.

Summary of Verified Featuresโ€‹

FeatureTestExpected BehaviourProof
Topology-aware scheduling enabledHelm install flag + scheduler logsScheduler uses topology-aware policyLog: --gpu-scheduler-policy=topology-aware
Single-GPU topology scoringNo extra flag - same topology-aware policy from Step 3Single-GPU Pods respect scores automaticallyPod annotation shows GPU7 UUID
Multi-GPU avoids isolated GPUPod requesting 2 GPUsGPU7 not allocatedAnnotation shows only GPUs from 0-6
Single-GPU picks isolated GPUPod requesting 1 GPUGPU7 allocated (when annotation persists)Annotation shows GPU7 UUID
Scheduler evaluates topology scoresVerbose (-v=6) logsSelection reflects your injected low score for GPU7Log: best device combination / worst device

Cleanupโ€‹

Delete the test Pods:

kubectl delete pod multi-gpu single-gpu --ignore-not-found

Uninstall HAMi and nvml-mock:

helm uninstall hami -n kube-system
helm uninstall nvml-mock

To stop the Kubernetes cluster, disable it from the OrbStack UI, or quit OrbStack entirely.

tip

If you want to keep the environment for further experimentation, skip the cleanup steps.

Next Stepsโ€‹

  • Move to a real GPU cluster to test actual memory/compute isolation with topology awareness.
  • Experiment with different topology matrices (e.g., 2 groups of 4 GPUs) and observe scheduling preferences.
  • Integrate this lab into a CI pipeline for regression testing of HAMi's scheduler.
CNCFHAMi is a CNCF Incubating project